---
title: "Two Vision Encoders Can Match Five in VLMs, Researchers Find"
subtitle: "A new study retrained all 31 encoder subsets from scratch, revealing that standard encoder evaluation methods are fundamentally misleading -- and that two encoders can match five."
url: "https://alphasignal.ai/news/two-vision-encoders-can-match-five-in-vlms-researchers-find"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-06-02T00:00:00.000Z"
updated: "2026-06-10T16:00:50.470Z"
type: "Paper"
topics:
  - "Image"
  - "Llms"
subtopics:
  - "Vision Language"
  - "Vision Language"
upvotes: 1859
source_url: "https://arxiv.org/abs/2606.03879"
comments: 0
truncated: true
---

# Two Vision Encoders Can Match Five in VLMs, Researchers Find

> A new study retrained all 31 encoder subsets from scratch, revealing that standard encoder evaluation methods are fundamentally misleading -- and that two encoders can match five.

When building a vision-language model (VLM), one of the most consequential decisions is which visual encoders to use. Models like [Cambrian-1](https://cambrian-mllm.github.io/cambrian-1/) and Eagle stack multiple encoders -- CLIP for language-aligned semantics, ConvNeXt for high-resolution detail, Pix2Struct for OCR, SAM for segmentation, EVA-02 for robust visual representations -- on the assumption that more encoders mean better performance. Combining multiple vision encoders to leverage their distinctive representations is a common strategy for building more capable multimodal models. But a new paper asks a harder question: do we actually understand what each encoder is contributing, and are we measuring it correctly?

## The flaw hiding in plain sight

The standard way to evaluate an encoder's importance is simple: take a fully trained multi-encoder checkpoint, mask out one encoder, and measure how much performance drops. It's cheap, fast, and widely used. The problem, according to this new work, is that it's wrong.

When encoders are masked on a fixed checkpoint versus retrained from scratch without them, performance gaps can diverge substantially -- sometimes by several percentage points -- and the answer generated by the model changes dramatically once a specific encoder is removed. The reason is straightforward: when you train a model with five encoders, the whole system adapts to that configuration. The projector (the small network that maps visual features into the language model's space) learns to expect all five streams. Masking one at inference time doesn't tell you what the model would have learned with four.

To fix this, the authors retrained every non-empty subset of five common encoders -- EVA-02, CLIP, ConvNeXt, Pix2Struct, and SAM -- under a unified pipeline. That's all 31 possible combinations, each trained from scratch. The best combination of vision experts evaluated in prior work includes CLIP, ConvNeXt, SAM, Pix2Struct, and EVA-02\. The total compute bill: roughly 20,000 GPU-hours.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs](https://arxiv.org/abs/2606.03879)
- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://cambrian-mllm.github.io/cambrian-1/)
- [Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders](https://arxiv.org/pdf/2507.03262)
- [Vision Encoders in Vision-Language Models: A Survey](https://jina.ai/vision-encoder-survey.pdf)
- [Papers Explained: Eagle (mixture-of-vision-encoders)](https://ritvik19.medium.com/papers-explained-269-eagle-09c21e549395)