Skip to main content
Back

Google DeepMind's TIPSv2 Fixes the Spatial Blind Spot Holding Vision AI Back

Google DeepMind's TIPSv2 fixes a long-standing weakness in vision-language models by supervising every patch, jumping zero-shot segmentation by 14.1 mIoU.

Pro

  • Google DeepMind released TIPSv2, a family of image-text encoders with strong patch-text alignment.
  • iBOT++ extends the masked image modeling loss to visible patches, gaining +14.1 mIoU on ADE150 zero-shot segmentation.
  • Head-only EMA cuts trainable parameters by 42% while preserving downstream performance.
  • Multi-granularity captions mix alt-text, PaliGemma, and Gemini Flash descriptions to avoid keyword shortcuts.
  • Beats DINOv3 on 4 of 6 shared evals despite DINOv3 using 15x more training images.
  • Code and four checkpoint sizes (B/14 to g/14) released under Apache 2.0 on Hugging Face.

Vision-language encoders like CLIP and SigLIP are great at telling you what is in a picture, but they are surprisingly bad at telling you where. Ask them to line up individual image patches with the words that describe those regions and the alignment falls apart, which limits their usefulness for segmentation, depth prediction, and any task that needs dense spatial understanding. TIPSv2, a new family of image-text encoders from Google DeepMind, targets that weakness and delivers state-of-the-art zero-shot segmentation while matching or beating recent encoders on classification and retrieval.

The paper started from an odd observation. A smaller ViT-L model distilled from a larger ViT-g TIPS teacher dramatically outperforms its teacher in zero-shot segmentation, reversing the trend on every other evaluation task. Distilled students beating teachers at one specific capability is unusual, and the authors treated it as a clue. After ablating masking ratios, initialization, frozen parameters, and supervision signals, they landed on the culprit: during distillation the student receives a training signal on visible patches, and during standard pretraining it does not.

The visible-token blind spot

Modern vision-language pretraining leans heavily on masked image modeling (MIM), and specifically on iBOT, where you hide some patches and train the model to reconstruct their representations. That works for the masked tokens, but the visible ones, which are the majority, get no direct patch-level supervision. Their representations drift toward whatever the global contrastive loss needs, and dense patch-text alignment suffers.