Frozen Qwen2.5-VL Beats Trained Image Search Models Without a Single Gradient Step
A new paper turns off-the-shelf multimodal LLMs into zero-shot image-to-image re-rankers, beating task-specific models on out-of-domain retrieval.
ProTakeaways
- Frozen Qwen2.5-VL-7B used as a zero-shot re-ranker via binary yes/no next-token probability on image pairs
- Outperforms task-specific AMES and fine-tuned LamRA on ILIAS, INSTRE, and Product1M without any training
- Reaches 54.4 mAP on ILIAS vs 41.9 for AMES and 33.4 for the no-re-rank global baseline
- Product Quantization indexing cuts per-image memory from ~2MB to kilobytes with minor accuracy loss
- Robust to clutter, occlusion, small objects; weak against illumination changes and motion blur
- Code at github.com/gkordo/mllm-ilr, evaluated on ILIAS benchmark
Multimodal LLMs are usually pitched as chat interfaces for images. A new paper from Mohamed bin Zayed University of AI and Czech Technical University in Prague flips that framing: it uses a frozen Qwen2.5-VL-7B as a training-free similarity engine for instance-level image retrieval, and shows it can outperform re-rankers built specifically for the job, without a single gradient step.
The mechanism is disarmingly simple. Prompt the model with paired images, ask whether they show the same object instance, and require a single-digit reply. The softmax probability assigned to the token "1" becomes the similarity score. That is the whole re-ranker.
Why bother with an LLM for pixels?
Instance-level retrieval is the fine-grained variant of image search: not "find another dog," but "find this exact dog." State-of-the-art systems typically bolt a re-ranking stage onto a global-descriptor search, using methods like RRT, CVNet, or AMES that cross-compare local patch features. They work, but they need domain-specific training data and tend to collapse outside their training distribution. AMES, for instance, is trained on the Google Landmarks dataset and excels at buildings while faltering elsewhere.