Two Vision Encoders Can Match Five in VLMs, Researchers Find
A new study retrained all 31 encoder subsets from scratch, revealing that standard encoder evaluation methods are fundamentally misleading -- and that two encoders can match five.
- Masking encoders on a fixed checkpoint gives wrong rankings -- retraining each subset from scratch reverses which encoder ranks first overall.
- Capacity vs. Necessity: a new two-axis decomposition separates how well an encoder performs alone from how much the full model needs it.
- Two encoders can match five: pairing a high-Capacity anchor with a high-Necessity complement nearly closes the gap to the full five-encoder pool.
- Pre-projector effective rank explains residual performance variation -- best pairs have an anchor whose rank survives joint training and a complement whose rank expands.
- 20,000 GPU-hours spent retraining all 31 non-empty subsets of 5 encoders (EVA-02, CLIP, ConvNeXt, Pix2Struct, SAM) on the Cambrian-1 16-benchmark suite.
- No code released, but the framework provides actionable primitives for encoder selection before committing to expensive training runs.
When building a vision-language model (VLM), one of the most consequential decisions is which visual encoders to use. Models like Cambrian-1 and Eagle stack multiple encoders -- CLIP for language-aligned semantics, ConvNeXt for high-resolution detail, Pix2Struct for OCR, SAM for segmentation, EVA-02 for robust visual representations -- on the assumption that more encoders mean better performance. Combining multiple vision encoders to leverage their distinctive representations is a common strategy for building more capable multimodal models. But a new paper asks a harder question: do we actually understand what each encoder is contributing, and are we measuring it correctly?
The flaw hiding in plain sight
The standard way to evaluate an encoder's importance is simple: take a fully trained multi-encoder checkpoint, mask out one encoder, and measure how much performance drops. It's cheap, fast, and widely used. The problem, according to this new work, is that it's wrong.
When encoders are masked on a fixed checkpoint versus retrained from scratch without them, performance gaps can diverge substantially -- sometimes by several percentage points -- and the answer generated by the model changes dramatically once a specific encoder is removed. The reason is straightforward: when you train a model with five encoders, the whole system adapts to that configuration. The projector (the small network that maps visual features into the language model's space) learns to expect all five streams. Masking one at inference time doesn't tell you what the model would have learned with four.
To fix this, the authors retrained every non-empty subset of five common encoders -- EVA-02, CLIP, ConvNeXt, Pix2Struct, and SAM -- under a unified pipeline. That's all 31 possible combinations, each trained from scratch. The best combination of vision experts evaluated in prior work includes CLIP, ConvNeXt, SAM, Pix2Struct, and EVA-02. The total compute bill: roughly 20,000 GPU-hours.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
