DOT-MoE Cuts Inference Costs 50% While Retaining 90% of Model Performance
DOT-MoE uses differentiable optimal transport to convert any dense LLM into a sparse MoE, cutting active parameters by 50% while retaining 90% of performance.
Takeaways
- DOT-MoE converts any dense LLM into a sparse MoE by framing neuron assignment as a differentiable optimal transport problem, accepted at ICML 2026.
- It jointly learns which neurons belong to which expert AND how tokens are routed, unlike prior methods that treat these as separate steps.
- Retains 90% of dense model performance while cutting active parameters by 50%, outperforming all structured pruning and MoEfication baselines.
- Alignment takes under 3 hours on 8×H100 GPUs for a 7B model; only assignment logits and router weights are trained — base model weights stay frozen.
- Scales to 32B parameters with a +34.3 point improvement over the best competing method (CMoE) on commonsense benchmarks.
- No throughput penalty from fine-grained experts when using fused MoE kernels (e.g., vLLM); the resulting model is a standard MoE compatible with existing serving frameworks.
Converting a dense language model into a sparse Mixture-of-Experts (MoE) without retraining from scratch is one of the most practical compression tricks available today. The idea, called MoEfication, lets you take a model like LLaMA-3 8B and turn its feed-forward layers into a set of expert sub-networks, then only activate a fraction of them per token at inference time. The catch: every existing method for deciding which neurons go into which expert relies on heuristics , random splits, weight clustering, or activation patterns , and those heuristics leave a lot of performance on the table. A new paper accepted at ICML 2026, authored by Udbhav Bamba, Arnav Chavan, Aryamaan Thakur, Steve Teig, and Deepak Gupta, proposes a principled fix.
Their framework, DOT-MoE, reframes the neuron assignment problem as a Differentiable Optimal Transport (DOT) problem , borrowing a mathematical tool from logistics and probability theory to find the best possible grouping of neurons into experts, end-to-end, via gradient descent.
Why heuristics fail
The core challenge in MoEfication is neuron assignment: determining how to partition the thousands of intermediate neurons in a dense FFN into discrete, independent, and functionally coherent experts. Existing approaches rely largely on heuristic strategies for neuron assignment, and they often treat neuron assignment and router training as separate processes. This two-stage approach creates a fundamental mismatch: the expert grouping is frozen before the router ever sees it, so the two can never co-adapt.
The deeper issue is what the authors call the assignment-routing gap. Existing approaches optimize proxies for intermediate representations while neglecting the actual output. Methods relying on input-side statistics or intermediate activations incur mean squared errors ranging from 2x to over 41x higher than the proposed approach. Clustering neurons by how similar their weights look tells you nothing about how those neurons jointly contribute to the final FFN output , which is what actually matters for downstream task performance.
Optimal transport as a neuron assignment engine
Optimal transport (OT) is a classical mathematical framework for moving mass from one distribution to another at minimum cost. Think of it as the most efficient way to redistribute sand from one set of piles into another. DOT-MoE repurposes this idea: neurons are the "mass" to be transported, and experts are the "destinations." Each neuron must go to exactly one expert, and each expert must receive exactly the same number of neurons , a strict balance constraint that prevents any expert from becoming a catch-all.