Peking University Finds 5% of AI Heads Do All the Visual Thinking
A new paper identifies a tiny subset of attention heads in multimodal LLMs that do all the visual retrieval work — and shows you can use that to cut inference time in half.
- Researchers identify "CoRe heads" — a sparse subset of attention heads in multimodal LLMs that handle nearly all cross-modal visual retrieval.
- A new metric called Retrieval Attention Mass (RAM) scores each attention head by how precisely it routes query attention to ground-truth visual tokens.
- Ablating just the top 5% of CoRe heads collapses multimodal reasoning accuracy; removing lower-ranked heads has almost no effect.
- ~30 specific heads remain consistently activated across image, video, and document tasks, showing a universal retrieval circuit.
- A hybrid attention strategy (dense for CoRe heads, sparse windows for the rest) achieves up to 2.3× prefill speedup with minimal accuracy loss.
- Code is open-source; tested on Qwen3-VL (4B/8B/32B), LLaVA-OneVision, and InternVL3.5.
When a multimodal model answers a question about a video or a document, it has to fish the right visual detail out of a sea of irrelevant pixels. Researchers from Soochow University and Peking University have now published a paper revealing exactly how that happens , and the answer is surprisingly concentrated. A tiny, specialized subset of attention heads does almost all the work, while the vast majority of heads just spread attention around without contributing meaningfully to the answer.
The problem nobody had mapped
Real-world visual inputs are often cluttered with redundant information, and robust multimodal reasoning depends on the model's capacity to selectively retrieve sparse, task-relevant visual cues from complex spatiotemporal scenes. Yet the precise internal mechanisms governing this cross-modal information retrieval remained unexplored, leaving a significant gap in our understanding of multimodal mechanistic interpretability.
Prior work on interpretability of vision-language models had mostly relied on coarse heuristics like spatial entropy or aggregate attention mass, and had been tested only on simple static images. None of it could explain fine-grained, query-conditioned retrieval in dense video or document contexts.
Introducing CoRe heads and the RAM metric
The researchers introduce a token-level metric called Retrieval Attention Mass (RAM) to identify a highly specialized subset of attention heads they call Context-aware Retrieval (CoRe) heads. Across diverse visual domains and model scales, a clear functional division emerges: CoRe heads act as dedicated information extractors, while most other heads distribute attention over broader contextual regions.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
