PRISM Merges CLIP, DINOv2 and SAM Into One Model, Beats All Three

There are three vision models that every serious computer vision practitioner knows: CLIP for semantic understanding, DINOv2 for fine-grained texture and correspondence, and SAM for precise geometric and boundary cues. Each excels at something the others miss. The obvious question is: can you get all three in one model? The obvious answer has always been "yes, but at a cost." That cost is negative transfer -- when you force a single network to satisfy contradictory training signals, the gradients from different teachers fight each other, and the model collapses into a mediocre average that is worse than any individual teacher at its specialty.

A new paper, PRISM: Synergizing Vision Foundation Models via Self-organized Expert Specialization, proposes a principled fix. Rather than cramming all knowledge into shared weights, PRISM uses a sparse Mixture-of-Experts (MoE) architecture that lets different experts specialize in different representational subspaces, then dynamically recombines them for each downstream task. The result is a new state of the art on two standard multi-task dense prediction benchmarks.

The gradient war inside your model

To understand why this matters, consider what happens when you distill CLIP and DINOv2 simultaneously into a single ViT backbone. Modern VFMs provide complementary cues -- high-level semantics from CLIP, spatial structures from SAM, and fine-grained texture or correspondence from DINOv2 -- but deploying them as a model ensemble is costly in memory, latency, and engineering complexity. So the natural move is to distill all three into one student. The problem is that this many-to-one distillation risks eliminating the representation biases of the VFM teachers, and reconciling diverse biases in shared parameters can induce optimization conflicts.

Concretely: DINO encourages feature variance to distinguish local textures, while CLIP suppresses that same variance to achieve semantic invariance. When both loss signals flow through the same weights, their gradients point in opposite directions. The aggregate update is a compromise that satisfies neither objective. This is the gradient conflict problem, and it gets worse as you add more teachers.

Prior work like SAK (Swiss Army Knife) addressed this with static architectural branching -- a shared stem plus teacher-specific adapter paths. Recent works distill multiple VFM teachers into a single student model, but this many-to-one distillation risks eliminating the representation biases of the VFM teachers, potentially limiting the model's ability to capitalize on their individual strengths for specific tasks. SAK's hard boundaries also create a different problem: visual knowledge has soft overlaps. CLIP and DINO both encode a representation of a cat, just at different frequency bands. Static partitioning ignores this nuance and creates parameter redundancy.