TaDA Merges Two AI Adapters Into One Without Any Retraining
A new training-free algorithm exploits a depth-dependent asymmetry in transformer layers to merge task and domain LoRA adapters into a single, better-performing model.
Takeaways
- Researchers propose TaDA, a training-free algorithm for merging a task LoRA and a domain LoRA into a single adapter with zero inference overhead.
- Key insight: task adapters dominate in early transformer layers; domain adapters dominate in later layers -- a pattern consistent across both language and vision models.
- TaDA uses probe-guided per-layer gating to assign different task/domain weights to each layer, plus SVD-based subspace filtering to discard conflicting directions.
- On 6 scientific QA benchmarks with Llama-2-7B, TaDA scores 45.2% average accuracy, beating DARE-TIES by +3.6 percentage points and winning all 6 benchmarks.
- On 6 image classification benchmarks with ViT-L/16, TaDA reaches 85.9% average accuracy, outperforming all merging baselines.
- No code repository is publicly linked yet; the paper is available in full on arXiv with sufficient detail for reimplementation.
A new paper published on arXiv introduces TaDA (Task-Domain LoRA Merging), a training-free algorithm for combining two distinct types of LoRA adapters -- one fine-tuned for a specific task, another for a specific domain -- into a single unified model. The work, by Huy Quoc To, Fuyi Li, Guangyan Huang, and Ming Liu, challenges a core assumption baked into every existing merging method: that both adapters should be treated equally across all layers.
The problem with treating adapters as equals
LoRA (Low-Rank Adaptation) has become the dominant way to fine-tune large models cheaply. Instead of updating all model weights, LoRA injects small trainable low-rank matrices into frozen model weights, cutting trainable parameters by orders of magnitude. In practice, teams often end up with two separate adapters: one trained on a downstream task (e.g., question answering), and one trained on a domain corpus (e.g., biomedical literature). Running both simultaneously is expensive. Merging them into one is the obvious solution -- but it turns out to be surprisingly hard to do well.
Combining a task LoRA adapter with a domain LoRA adapter into a single unified model is a practical yet largely unexplored challenge, and existing methods treat both adapters as symmetric peers, applying uniform weights across all layers. Methods like TIES and DARE-TIES -- which resolve parameter conflicts by trimming small values and resolving sign disagreements -- were designed for merging multiple task adapters together. The paper's results demonstrate that existing methods designed for task-to-task merging are inadequate for this asymmetric setting.
A structural insight hiding in plain sight
The core contribution of TaDA is an empirical observation that turns out to hold consistently across both language and vision transformers: task and domain adapters exhibit a consistent depth-dependent asymmetry across transformer architectures -- domain dominance increases with layer depth, while shallower layers retain stronger task-relevant signals.