HARC Fixes the Broken Router That Ruins AI Model Merging
HARC fixes a critical but overlooked failure mode in MoE model merging, using second-order math to recalibrate broken routers without any retraining
Takeaways
- HARC is a training-free method that fixes "routing breakdown" -- a critical failure mode when merging MoE language models.
- Standard merging techniques (TIES, Task Arithmetic, DARE) alter expert weights but leave the router misaligned, causing tokens to be sent to wrong experts.
- HARC uses second-order Hessian curvature information to compute a closed-form correction to the router weights after merging.
- A matrix-free conjugate gradient solver makes the approach efficient -- no full Hessian matrix is ever built.
- Experiments on math reasoning and code generation show substantial performance recovery across multiple merging baselines including WUDI and RegMean.
- Code is available on GitHub and works as a drop-in post-processing step for any MoE merging pipeline.
Model merging has become one of the most compelling tricks in the modern LLM toolkit. Instead of training a new model from scratch, you take two or more fine-tuned models and arithmetically combine their weights, getting a multi-skilled model for nearly free. Merged models have even reached top rankings on competitive leaderboards including the Open LLM Leaderboard, with strategic model combination yielding capabilities exceeding those of individual fine-tuned variants. But there is a catch that nobody has cleanly solved: this approach breaks badly when applied to Mixture-of-Experts (MoE) architectures. A new paper introduces Hessian-Aware Router Calibration (HARC), a training-free fix that targets the root cause of this failure.
Why MoE Merging Is a Different Beast
Before getting into the fix, it helps to understand what makes MoE models special. In a standard dense model, every parameter is active for every input. The Mixture-of-Experts architecture controls computational cost while scaling parameters: input tokens are routed to only a small subset of experts, trading higher memory overhead for lower computational cost. The router is the gatekeeper -- a small learned network that looks at each token and decides which experts should handle it.
MoEs employ a learned routing function to dynamically select a subset of experts for each input, achieving conditional computation by activating only relevant model components. This input-adaptive selection promotes expert specialization and modular knowledge reuse. That specialization is exactly what makes MoE models powerful, and exactly what makes merging them so dangerous.
The recent success of specialized LLMs in domains such as mathematical reasoning and coding has led to growing interest in methods for merging these expert LLMs into a unified MoE model, with the goal of enhancing performance in each domain while retaining effectiveness on general tasks. The problem is that effective merging of expert models remains an open challenge, especially for models with highly divergent weight parameters. State-of-the-art MoE merging methods only work with homogeneous model architectures and rely on simple unweighted averaging to merge expert layers, which does not address parameter interference and requires extensive fine-tuning of the merged MoE to restore performance.