Skip to main content
Back

EPFL's FlexMerge Recovers 13.5% Accuracy by Rethinking AI Model Merging

FlexMerge lets you dial the merged model size from 1x to full, revealing 13.5% accuracy jumps just by doubling storage.

Pro

  • FlexMerge generates data-free merged models at any size between 1x and M x fine-tuned checkpoints.
  • Doubling merged size from 1x to 2x delivers up to 13.5% accuracy gains on vision benchmarks.
  • Algorithm rankings change with size; simpler methods often beat sophisticated ones at larger sizes.
  • Greedy pairwise block merging uses minimum cosine similarity between task vectors as the criterion.
  • Near fine-tuning accuracy reached at 6x for 8 tasks, 23.5x for 30 tasks.
  • Code and full framework available at github.com/sacs-epfl/flexmerge.

Model merging has become a popular way to combine many single-task fine-tuned checkpoints into one multi-task model without retraining. The catch is that squeezing everything into a single set of weights leaves a noticeable accuracy gap compared to the original fine-tuned models, and that gap widens as more tasks are added. A new framework called FlexMerge argues the field has been optimizing the wrong variable, and that a bit of extra storage buys a surprising amount of accuracy back.

The one-model bottleneck

Merging combines multiple fine-tuned checkpoints into a single model without access to training data, sidestepping the compute cost and privacy headaches of joint multi-task training. A single set of weights, though, often cannot resolve parameter conflicts between tasks, and the resulting accuracy gap grows as more models are folded in.

The obvious workaround, keeping every checkpoint on disk, blows up storage linearly with the number of tasks. Some methods narrow the accuracy gap by pulling in extra validation data, but that dependency is hard to satisfy under privacy constraints or proprietary restrictions. Data-free merging, where the accuracy hit is still very real, is what remains.

Merging as a spectrum, not a switch

The authors from EPFL and Carnegie Mellon reframe the problem by treating model size as a continuous knob rather than a choice between one fused model and M full checkpoints. At one extreme, retaining every fine-tuned model achieves maximal accuracy at the cost of larger overall size; at the other, fully merging into a single model minimizes storage but sacrifices accuracy. FlexMerge lets you stop anywhere in between, including at fractional sizes like 1.75x or 2.34x.