Skip to main content
Back

Stanford Exposes Why Vision AI Ignores Images, Offers a Training-Free Fix

A new probe shows text tokens dominate vision by 4x in multimodal models, and a simple contrastive decoding trick recovers up to 16.9% accuracy.

Paper, Full Access
  • Text tokens dominate vision by 4x across seven multimodal models on visual perception tasks.
  • Centroid replacement probe collapses tokens to their nearest K-means cluster to measure modal dependence.
  • Text centroid contrastive decoding recovers up to +16.9% accuracy without retraining, at 2x inference cost.
  • Standard SFT and DPO models gain +5.6% on average; heavily post-trained models only +1.5%.
  • The asymmetry replicates on VPBench, CV-Bench, MMStar, MMVP, and clinical MedBLINK.
  • Authors will release code and centroids; useful as post-deployment audit for modal balance.

Anyone who has tried to get a vision-language model to actually look at an image has run into the same wall: the model confidently answers based on what the question sounds like, not what the picture shows. A new paper from Stanford, University of Washington, UNC Chapel Hill, and USI Lugano puts a number on that failure mode and offers a training-free fix.

The authors propose centroid replacement, collapsing each token to its nearest K-means centroid, as a controlled probe for modal dependence. When they apply this identical intervention to visual tokens and text tokens separately, the asymmetry is stark. Across seven models spanning three architecture families, erasing text centroid structure costs 4x more accuracy than erasing visual centroid structure, exposing a universal imbalance where language representations overshadow vision even on tasks that demand visual reasoning.

Why vision loses to language

Multimodal language models chain a vision encoder to a language model backbone. Prior work has shown that the vision encoders themselves already contain enough information to solve visual perception tasks, but the language backbone fails to use it. Gradient analyses have found that language token features impact the loss over 10x more than visual features, and cross-modal attention scores skew heavily toward text. Models score near-human on text-shaped benchmarks and stumble on depth estimation, forensic artifact detection, or spatial layout.

This paper contributes a geometric explanation. If you replace each hidden-state token with the centroid of the K-means cluster it belongs to, you are throwing away all the fine-grained within-cluster variation while keeping the coarse structure. Doing this to visual tokens barely hurts accuracy. Doing it to text tokens is catastrophic. Two of the seven models tested actually get slightly better when visual tokens are replaced with centroids, meaning within-cluster visual variation was mildly harmful in the first place.