---
title: "TAP Slashes AI Model Merging Costs by 1000x Without Decoder Training"
subtitle: "A new proxy method estimates model merging quality from feature similarity alone, cutting hyperparameter search time by up to 1000x."
url: "https://alphasignal.ai/news/tap-slashes-ai-model-merging-costs-by-1000x-without-decoder-training"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-04-14T00:00:00.000Z"
updated: "2026-08-23T22:57:03.577Z"
type: "Paper"
topics:
  - "Image"
  - "Post Training"
subtopics:
  - "Vision Language"
  - "Distillation"
  - "Fine Tuning"
upvotes: 4500
source_url: "https://arxiv.org/abs/2604.12935"
comments: 0
truncated: true
---

# TAP Slashes AI Model Merging Costs by 1000x Without Decoder Training

> A new proxy method estimates model merging quality from feature similarity alone, cutting hyperparameter search time by up to 1000x.

Model merging combines several fine-tuned models into a single one without extra training, which is attractive when you want to deploy a robot with a single encoder that handles segmentation, depth, pose, and mesh recovery instead of shipping four separate networks. The catch is that most published merging methods were validated on an unusually easy scenario: CLIP-based image classification, where the decoder is a frozen text encoder. A [new paper](https://arxiv.org/abs/2604.12935) introduces the Task Alignment Proxy (TAP), a lightweight signal that lets you pick merging hyperparameters without ever training a decoder, and pushes merging into settings that were previously impractical.

## Why the old evaluation loop breaks

In CLIP-based benchmarks the visual encoder is fine-tuned separately for each task while a frozen text encoder acts as a shared classifier, so evaluating a merged model only requires forward inference and exhaustive hyperparameter search is feasible. Real vision stacks look nothing like that. Most multi-task systems produce outputs in different spaces such as semantic labels, depth maps, 3D poses, or meshes, and rely on task-specific decoders attached to a shared encoder. Only the encoder is merged, while decoders remain task-specific and must be fine-tuned after merging.

That flips the economics of hyperparameter search. Every candidate merge needs a fresh decoder trained on top before you can measure anything. In the paper's LiDAR setup, decoder training requires more than 30 GPU hours per evaluated hyperparameter, and for the MapFree relocalization task with the MASt3R decoder, fine-tuning requires 4 H100s and takes around 4 days to complete. Sweeping tens of merge configurations that way is a non-starter.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Task Alignment: A simple and effective proxy for model merging in computer vision](https://arxiv.org/abs/2604.12935)
- [Task Alignment paper (HTML)](https://arxiv.org/html/2604.12935)
- [ScaLR: image-to-LiDAR distillation](https://github.com/valeoai/ScaLR)
- [Task Singular Vectors](https://github.com/AntoAndGar/task_singular_vectors)