Sapienza's MASS Merges 20 AI Models Into One Without Losing Accuracy
A new model merging method uses SVD subspaces and a data-free router to recover 98% of fine-tuned accuracy at roughly 2x storage.
ProTakeaways
- MASS closes the gap between merged and individually fine-tuned models, recovering up to 98% of accuracy.
- A data-free, training-free router picks task subspaces by projection residual on mid-layer activations.
- Storage stays around 2x a single backbone regardless of the number of merged tasks.
- Sets new state of the art on 8, 14, and 20 task CLIP benchmarks across ViT-B/32, ViT-B/16, ViT-L/14.
- Beats TwinMerging (97.6 vs 95.3 normalized) despite not assuming oracle head selection.
- Code, checkpoints, and configs are on GitHub with models on Hugging Face.
Fine-tuning a shared backbone for a dozen different tasks leaves you with a dozen full-size checkpoints and a nasty deployment problem. Model merging promised a way out by folding those checkpoints into a single set of weights, but every merging trick to date has left noticeable accuracy on the table. A team from Sapienza University of Rome is pushing that gap close to zero with a method called MASS (MoErging through Adaptive Subspace Selection), which behaves like a mixture of experts hiding inside a single merged model.
MASS unifies multiple fine-tuned models while retaining near state-of-the-art performance across tasks, and it does so without any training and without needing access to the original task data. On CLIP-based image classification benchmarks with 8, 14, and 20 tasks, it sets a new state of the art and, in the friendliest inference setting, essentially matches individually fine-tuned models.
Why merging keeps losing accuracy
The dominant recipe for merging today is task arithmetic. The weights of a multi-task model for T tasks are computed by aggregating the task-specific weight differences, or task vectors, meaning you take each fine-tuned checkpoint, subtract the pretrained weights to get a task vector, then add all task vectors back into the backbone with a scaling factor.
The problem is rigidity. Once you sum the vectors, the merged model treats every input the same way, even though a car photo and a handwritten digit clearly want different specialists. Structured methods like Task Singular Vectors (TSV) improved on this by exploiting the low-rank structure of per-layer weight deltas, but the aggregation is still fixed. The authors point out that TSV reports around a 60% accuracy drop for the worst-performing task compared to its fine-tuned endpoint. That is the wound MASS wants to close.
Keep the subspaces, activate them on demand
MASS builds on the observation that per-task weight updates are approximately low rank, so each task can be summarized by a small set of singular vectors. Instead of collapsing all those vectors into one static sum, MASS keeps them as distinct, largely orthogonal subspaces sitting inside a single merged model, and then decides at inference time which subspaces to switch on for the current input.