Meta FAIR Proves Dedicated Vision Capacity Beats Pretrained Encoders by 13%
A controlled study of 200+ multimodal models from scratch finds that vision needs its own parameters, but neither separate pretraining nor a dedicated encoder.
- Meta FAIR and NYU trained 200+ multimodal models from scratch to isolate three vision encoder design choices.
- Separate encoder pretraining (CLIP, SSL, captioning) offers no VQA benefit at matched lifetime compute, even when free.
- Encoder-free models trail by 4-7 Avg. VQA points and need ~3x compute to catch up, even with parameters refunded.
- Dedicated vision capacity is what matters; placement before the LLM is not required.
- In-fusion vision-only modules in the first 8 backbone layers match the encoder (45.0 vs 44.4 Avg. VQA).
- In-fusion cuts time to first token by 13% at 0.4B params. Project, paper.
Most multimodal language models attach a pretrained vision encoder such as SigLIP or CLIP to a language backbone. Newer systems, including Kimi K3, train the encoder jointly with the language model, while encoder-free designs convert image patches directly into tokens for the backbone. Comparisons among these approaches have been inconclusive because published models also vary in training data, architecture, and compute.
Researchers from Meta FAIR and NYU addressed those confounders in Unbundling Multimodal Vision Encoding. They ran more than 200 controlled training experiments using common data and matched lifetime compute. Their results attribute the encoder’s gains to dedicated capacity for image tokens. Separate vision pretraining provides no consistent advantage, and the dedicated parameters can sit inside the language backbone instead of a standalone encoder.
Three Jobs Bundled Together
A conventional pretrained vision encoder combines three architectural choices that earlier comparisons rarely isolate:
- Visual prior: knowledge learned during a separate stage such as CLIP-style contrastive training or self-supervised vision training.
- Dedicated vision capacity: parameters reserved for image tokens and unavailable to text tokens.
- Pre-fusion placement: image processing that happens before visual and text tokens enter the shared model.
The study separates those choices across four model recipes:
| Design | Visual prior | Dedicated capacity | Placement |
|---|---|---|---|
| Pretrained encoder | Yes | Yes | Before the backbone |
| Scratch encoder | No | Yes | Before the backbone |
| Encoder-free | No | No | Image patches enter the backbone |
| In-fusion | No | Yes | Inside early backbone layers |
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
