ID Balancing Cuts Routing Overload by 89% in 70B Sparse AI Models

Training a highly sparse Mixture-of-Experts model becomes harder as the expert pool grows and the number selected per token shrinks. A small routing skew can send excess traffic to a few experts while leaving others undertrained. A new paper introduces ID Balancing, a control-theory approach that stabilizes routing in configurations reaching 768 experts and 69.9 billion parameters.

Why extreme sparsity destabilizes routing

Mixture-of-Experts (MoE) layers increase parameter count without a proportional increase in per-token computation. A learned router scores every expert, then sends each token to only the top K. In a Top-3-of-768 configuration, for example, each token activates 0.4 percent of the available experts.

That efficiency depends on distributing tokens evenly. Popular experts can exceed their processing capacity and drop assignments, while lightly used experts receive too little training. In expert-parallel systems, where experts run on separate accelerators, overloaded workers delay the entire step as other devices sit idle. Persistent concentration can develop into routing collapse.

Auxiliary load-balancing losses address the problem by penalizing uneven assignments, but their gradients modify the language-modeling objective. DeepSeek-V3 instead adjusts a per-expert router bias outside that objective: overloaded experts receive a lower bias, and underused experts receive a higher one. Kimi’s Quantile Balancing uses a related bias update. The paper finds that both approaches lose effectiveness as routing becomes more sparse.

Router bias becomes a feedback loop

The paper models bias-based routing as a feedback controller. The system measures each expert’s load, compares it with the target average, updates the routing bias, and observes the resulting distribution during later steps. The bias therefore acts as controller state rather than a learned parameter updated through backpropagation.