Skip to main content
Back

KAIST's AIF Fixes Vision AI's Seeing-Without-Perceiving Bug Without Retraining

A training-free trick reshapes the causal mask at inference time, forcing vision-language models to ignore distracting image regions and answer more accurately.

  • KAIST and POSTECH show that VLMs often see the right region but attend to distractors, causing wrong answers.
  • Adaptive Information Flow (AIF) is a training-free method that rewrites the causal mask at inference time.
  • Per-layer attention entropy identifies important visual tokens; text tokens are blocked from attending to noisy ones.
  • Boosts LLaVA-1.5-7B and Qwen2.5-VL-7B across VQA, OCR, grounding, counting, and POPE hallucination.
  • Qwen2.5-VL-7B + AIF averages 88.9 on RefCOCO, reportedly beating Grounding-DINO-L without retraining.
  • Code planned at kaist-ami/AIF; overhead is roughly one extra decoding step.

Vision-language models have a strange failure mode: they often look at exactly the right part of an image, then still get the answer wrong. A CVPR 2026 paper from KAIST and POSTECH pinpoints the culprit and offers a fix that requires no retraining, no extra data, and almost no extra compute. The method, called Adaptive Information Flow (AIF), changes which visual tokens the text tokens are allowed to attend to during decoding.

The core observation is unsettling if you build with VLMs. Recent work shows that while these models often capture the correct image region for a given question, they do not necessarily produce the correct answer. The authors trace this seeing-versus-perceiving gap to how attention gets distributed inside the language model backbone.

Attention that leaks everywhere

In a standard VLM forward pass, visual and text tokens are concatenated and pushed through a decoder-only LLM under a causal mask. The problem the authors document is diffusion: instruction tokens spread their cross-attention across many background visual tokens instead of concentrating on the object that actually answers the question. That dispersion pumps noisy visual information into the textual decoding path and pushes the model toward wrong answers.

Diagram showing text-to-image attention on a Mickey Mouse clock and activation patterns across LLM layers

To probe this, the team ran an oracle experiment on LLaVA-1.5-7B. They defined a per-token statistic they call token dynamics: for each visual token, the maximum attention it receives from any text token, tracked across every layer of the LLM. Averaging those values per token yields a scalar importance proxy. The authors then manually masked visual tokens by that score and watched the accuracy curves.