Basque Country Researchers Prove CLIP Already Understands Composition but Queries It Wrong

For years, dual-encoder vision-language models like CLIP have been dismissed as glorified bag-of-words systems. Ask one to distinguish between a black dog and a white cat and a black cat and a white dog, and it often shrugs. The prevailing assumption has been that the encoders themselves fail to capture compositional structure. A new paper from the HiTZ Center at the University of the Basque Country argues that the encoders are largely fine, and that the failure lives in what we do with their outputs.

The authors show that the standard practice of comparing pooled image and text embeddings with a single cosine similarity discards most of the compositional information already sitting in patch and token representations. Fix the inference procedure, and compositional accuracy jumps sharply without touching a single encoder weight.

Where the bottleneck actually lives

Compositional reasoning means correctly binding objects to their attributes and relations across modalities. Standard evaluations of dual-encoder VLMs rely on global cosine similarity between pooled image and text embeddings, without explicitly modeling interactions between image regions and textual components. That single scalar collapses everything the model knows about which patch corresponds to which token.

To test whether the encoders are truly deficient or just poorly queried, the authors designed a diagnostic inference protocol called Structure-Guided Inference (SGI). The pipeline is deliberately simple and requires no training:

1. Cut the image into a bank of crops at multiple scales and aspect ratios (roughly 270 crops with overlap).
2. Parse the caption into object-attribute phrases plus the full sentence using deterministic rules or spaCy noun chunks.
3. Embed every crop and every text segment with the frozen encoders and build an N x M cosine similarity matrix.