Skip to main content
Back

ProbMoE Fixes the Broken Gradient Problem Holding Back MoE Models

ProbMoE replaces top-k's broken gradients with probabilistic subset sampling, delivering better expert diversity and adaptive compute per token

Paper, Full Access
  • Core problem: Top-k routing in MoE models is non-differentiable, forcing biased gradient hacks like the straight-through estimator during training.
  • ProbMoE's fix: Models expert selection as a distribution over k-expert subsets; backward pass uses each expert's exact marginal probability as a low-bias gradient surrogate.
  • Two variants: Exact-k (drop-in replacement, same inference cost, better training) and Dynamic-k (adaptive experts per token, fewer active experts at inference).
  • Results: Exact-k improves expert utilization and routing diversity; Dynamic-k matches fixed top-k quality with fewer activated experts.
  • Key advantage over alternatives: Unlike Soft-MoE or ReMoE, ProbMoE keeps hard discrete selection in the forward pass, staying compatible with standard MoE inference stacks.
  • Availability: No public code yet; builds on the SIMPLE gradient estimator; paper at arxiv.org/abs/2606.01509.

Mixture-of-Experts (MoE) models -- the architecture powering DeepSeek, Mixtral, and Qwen -- are built on a clever premise: instead of running every parameter for every token, route each token to a small subset of specialized sub-networks called experts. The efficiency gains are real, but training these models has always carried a hidden tax: the routing step is fundamentally broken from a gradient perspective.

A new paper from researchers introduces ProbMoE, a probabilistic routing framework that attacks this problem at its root by reframing expert selection as probabilistic inference over discrete subsets -- and the results suggest it is a meaningful step forward for both training quality and compute efficiency.

The Gradient Problem Nobody Solved

Standard MoE routing works by scoring each expert for a given token, then picking the top-k highest scorers. The argmax operation at the heart of this is not differentiable -- backpropagation requires gradients, and gradients require continuity, but argmax produces a discrete integer index that jumps discontinuously as inputs change. This means the router cannot be trained end-to-end in the usual sense.

The conventional workaround is the straight-through estimator (STE) -- a trick where you use the hard discrete selection in the forward pass but pretend the operation was a softmax during the backward pass. The basic idea is to use two different computations for the forward and backward passes, accepting this inconsistency as a pragmatic compromise that enables training to proceed. It works well enough to train models like Mixtral and DeepSeek, but it introduces bias and can hurt expert specialization.

The downstream effects are real. Experts that lose routing competitions receive no gradient signal and no opportunity to refine their representation of that kind of input, creating strong selective pressure where experts must learn to be distinctively better than all others at some identifiable class of tokens. In practice, this often leads to uneven expert utilization -- some experts handle most tokens, others are nearly idle.