TAP Slashes AI Model Merging Costs by 1000x Without Decoder Training
A new proxy method estimates model merging quality from feature similarity alone, cutting hyperparameter search time by up to 1000x.
PRO- New arXiv paper introduces Task Alignment Proxy (TAP) to pick model merging hyperparameters without training decoders.
- TAP measures feature similarity between merged and fine-tuned encoders on ~128 unlabeled images per task.
- Cuts hyperparameter selection cost by up to 3 orders of magnitude on heterogeneous vision benchmarks.
- First demonstrated successful model merging on LiDAR semantic segmentation across four sensor types.
- Matches full validation-based selection on standard CLIP merging benchmarks (ViT-B/32, B/16, L/14).
- Analysis shows unbalanced task vector norms are the main open challenge for merging beyond CLIP.
Model merging combines several fine-tuned models into a single one without extra training, which is attractive when you want to deploy a robot with a single encoder that handles segmentation, depth, pose, and mesh recovery instead of shipping four separate networks. The catch is that most published merging methods were validated on an unusually easy scenario: CLIP-based image classification, where the decoder is a frozen text encoder. A new paper introduces the Task Alignment Proxy (TAP), a lightweight signal that lets you pick merging hyperparameters without ever training a decoder, and pushes merging into settings that were previously impractical.
Why the old evaluation loop breaks
In CLIP-based benchmarks the visual encoder is fine-tuned separately for each task while a frozen text encoder acts as a shared classifier, so evaluating a merged model only requires forward inference and exhaustive hyperparameter search is feasible. Real vision stacks look nothing like that. Most multi-task systems produce outputs in different spaces such as semantic labels, depth maps, 3D poses, or meshes, and rely on task-specific decoders attached to a shared encoder. Only the encoder is merged, while decoders remain task-specific and must be fine-tuned after merging.
That flips the economics of hyperparameter search. Every candidate merge needs a fresh decoder trained on top before you can measure anything. In the paper's LiDAR setup, decoder training requires more than 30 GPU hours per evaluated hyperparameter, and for the MapFree relocalization task with the MASt3R decoder, fine-tuning requires 4 H100s and takes around 4 days to complete. Sweeping tens of merge configurations that way is a non-starter.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
