Skip to main content
Back

Peking University's UltraEP Fixes DeepSeek-V3 Load Imbalance, Hitting 94.3% Peak Throughput

UltraEP solves real-time expert load imbalance in MoE training and inference, hitting 94.3% of theoretical peak throughput across models up to 671B parameters

Paper, Full Access
  • Real-time MoE load balancing: UltraEP rebalances expert load every microbatch and layer, unlike periodic historical-based approaches.
  • 1.49x throughput gain: Averaged across 106B-671B parameter models, UltraEP delivers 1.49x over no balancing and reaches 94.3% of theoretical peak.
  • Near-perfect balance: Reduces inter-rank imbalance ratio from 1.30-4.01 down to 1.01-1.04 across all tested models.
  • Three-part mechanism: Quota-driven planning, RSN-native persistent tile streaming, and relay-based fan-out mitigation work together on the critical path.
  • Production validated: Tested on RefMoE-288B with 2560 GPUs across 40 racks; benchmarked on DeepSeek-V3, Qwen3-235B, and GLM4.5/4.7.
  • RSN-specific, no code release: Requires rack-scale node hardware with high-bandwidth scale-up links; no public implementation available.

Mixture-of-Experts (MoE) models like DeepSeek-V3 and Qwen3 have become the dominant architecture for scaling frontier LLMs, but they carry a hidden operational tax: expert load imbalance. When tokens flood into a handful of popular experts while others sit idle, the entire cluster slows to the pace of the most overloaded GPU. A new paper from researchers at Peking University and collaborators introduces UltraEP, a real-time load balancer that nearly eliminates this problem without waiting for the next training checkpoint to fix it.

The Straggler Problem Nobody Talks About

MoE models have become a dominant paradigm for scaling LLMs, offering high model capacity with sparse activation. The key idea is that only a small subset of "expert" sub-networks fires for any given token, keeping compute costs manageable even as total parameter counts balloon. To accommodate larger MoE models, expert parallelism (EP) is widely adopted, where experts are distributed across devices and tokens are dynamically routed via all-to-all communication.

The problem is that routing is never perfectly uniform. Large-scale EP amplifies device-level expert load imbalance into compute stragglers, token all-to-all bottlenecks, and activation-memory spikes. Think of it like a restaurant where every customer keeps ordering from the same two chefs while six others stand around. The whole kitchen stalls waiting for those two. Even well-trained MoE models exhibit significantly imbalanced routing -- and this is arguably by design, since concentrated expertise can improve model quality.

In production, large-scale expert parallelism, such as 32- or 64-way EP, has been indispensable for training and serving MoE models with hundreds of billions of parameters. At that scale, even modest imbalance compounds into serious throughput loss. The existing fix -- EPLB (Expert Parallel Load Balancing) used in systems like SGLang -- works by periodically reshuffling expert placement based on historical load statistics. Existing balancers redistribute experts periodically based on historical load, which becomes unreliable for production deployments with non-stationary load patterns. In other words, by the time the rebalancer acts, the load distribution has already shifted.