V-GIFT Forces Vision AI to Actually Look at Images
A new paper shows that mixing just 3-10% of self-supervised visual puzzles into instruction tuning fixes language shortcuts in multimodal LLMs.
ProTakeaways
- V-GIFT boosts multimodal LLMs by injecting 3-10% self-supervised visual tasks as chat instructions during fine-tuning.
- Three pretext tasks used: rotation prediction, point colorization, and cross-view point correspondence.
- No new losses, no architecture changes, no extra training stages, minimal compute overhead.
- Gains up to +3.4 on BLINK for LLaVA-OneVision-1.5 and +3.9 CVB-2D under LoRA training.
- Attention maps confirm the model actually looks at task-relevant objects after V-GIFT training.
- Code and trained weights are open source on GitHub and Hugging Face.
Multimodal LLMs have a dirty secret: when you ask them to count lamps in a photo or judge which object is closer to the camera, they often ignore the pixels and guess from context. A new method called V-GIFT (Visually Grounded Instruction Fine-Tuning) tackles this with a change small enough to fit inside the existing training loop, with no new losses, no new modules, and no extra stages.
The trick is to sprinkle in a handful of self-supervised visual tasks, reformatted as natural-language instructions, that cannot be answered without looking at the image. On the LLaVA-1.5 and LLaVA-OneVision-1.5 backbones, adding these samples at a 3-10% ratio produces consistent gains on vision-centric benchmarks like CV-Bench 2D, POPE, MMStar, and BLINK.
The language shortcut problem
Modern vision-language models pair a strong image encoder (CLIP, DINOv2, SigLIP) with a pretrained LLM through a light projector. Recent probing work has shown that these encoders already carry the visual information needed for fine-grained reasoning. The bottleneck sits in the LLM decoder, which learns during instruction tuning to lean on linguistic priors rather than the visual tokens it is being fed.
The reason is baked into how instruction data is built. Most image-instruction-response triplets are auto-generated from captions, so many are answerable through language shortcuts. If a photo caption mentions a beach, the model can guess the answer to "what activities happen here?" without ever attending to the pixels. Training on many such examples reinforces the shortcut. The authors reframe this as a modality competition problem: when text priors are sufficient too often, the optimizer quietly biases the model toward the linguistic path.