STS Fixes LLaVA's Costly Visual Token Collapse With a Two-Stage Pruning Fix
A new two-stage pruning framework fixes the core flaw in attention-based token selection, boosting both spatial coverage and task-relevant precision in VLMs.
Takeaways
- Core problem: Attention-based visual token pruning in VLMs causes "attention collapse" -- retaining redundant, spatially clustered tokens while discarding useful context.
- New method: Structure-to-Semantics (STS) is a two-stage framework that first maximizes spatial diversity, then filters by prompt relevance.
- Stage 1: Repulsion-based sampling spreads selected tokens across the full image geometry, preventing spatial clustering.
- Stage 2: Instruction-aware cross-attention removes tokens that survived Stage 1 but are irrelevant to the user's query.
- Results: STS outperforms LLaVA, LLaVA-NeXT, and FastMoE-LLaVA on GQA, POPE, and VQA benchmarks, especially at aggressive pruning ratios.
- Availability: No public code yet; the paper is training-free and designed for plug-and-play inference integration.
Vision-Language Models (VLMs) like LLaVA are powerful, but they come with a steep computational price. Visual tokens comprise the majority of the input sequence and significantly increase the computational cost of VLM inference -- LLaVA-1.5, for instance, converts a single 336x336 image into 576 visual tokens, while textual inputs typically stay under 100 tokens. Higher resolutions push this even further, with LLaVA-NeXT generating 2,880 visual tokens per image. The standard fix -- pruning the tokens that get the least attention -- turns out to have a fundamental flaw. A new paper proposes a smarter two-stage alternative.
The hidden cost of "just use attention"
VLMs rely on long visual token sequences for visual understanding, making the prefill stage expensive in both computation and memory. Most existing pruning methods follow an absolute-ranking paradigm: assign importance scores to visual tokens and retain a fixed top-K subset. The problem is that attention scores are not a neutral signal.
When token selection is based exclusively on attention scores, the model tends to retain similar clusters. Adjacent tokens with similar visual features frequently receive comparable attention scores, especially in regions with flat backgrounds or repetitive textures -- and their spatial proximity means they capture overlapping features, making it hard to distinguish the tokens that are not highlighted but still informative. In short, you end up keeping a redundant cluster of tokens that all say the same thing, while discarding the peripheral context that might actually matter for answering a question.
Attention in VLMs is also systematically biased: it disproportionately favors tokens appearing later in the sequence and assigns inflated scores to semantically empty padding tokens. These behaviors stem from recency bias and attention sink effects inherited from LLMs, and they distort attention-based pruning by preserving irrelevant visual content.