MIT Warns Muon Optimizer Silently Breaks in Late Transformer Layers at Scale
A new paper from MIT maps how Muon's momentum spectra scale with model size, revealing a hidden failure mode in late layers at frontier scale -- and how to fix it cheaply.
Takeaways
- Core finding: Momentum matrix singular values in Muon follow clean power laws in model size, with layer-dependent exponents -- the first systematic study of this behavior.
- Late-layer risk: Final MLP and output projection layers scale as aggressively as M^{-0.96}, pushing them into Newton-Schulz failure at frontier scale with the standard 5-step config.
- Most layers are fine: Mid-depth and attention layers (Q, K, V) scale only at ~M^{-0.25}, meaning 5-step NS remains accurate well beyond academic-scale models.
- Practical recipe: The paper gives a layer-aware guide for choosing the minimum NS iterations per layer type, avoiding wasted compute without sacrificing update quality.
- Industry context: Muon is now used in Kimi-K2 (1T params), GLM-5 (744B), and DeepSeek-V4 (1.6T), making its scaling behavior a production-critical question.
- No code released, but the methodology (tracking singular value quantiles + power-law fitting) is straightforward to add to any Muon training loop. Read the paper.
The Muon optimizer has gone from a NanoGPT speedrunning trick to a core ingredient in some of the most capable open models in the world. Zhipu AI's GLM-5 (744B parameters) confirmed the use of Muon in both GLM-4.5 and GLM-5 pretraining, and DeepSeek-V4 (1.6T parameters) also employs Muon for faster convergence and greater training stability. Kimi K2, a mixture-of-experts model with 32 billion activated parameters and 1 trillion total parameters, was trained with Muon and achieves exceptional performance across frontier knowledge, reasoning, and coding tasks. As Muon gets applied at ever-larger scales, a quiet question has been building: does the math that makes it work at 100M parameters still hold at 100B?
A new paper from MIT answers that question -- and the answer is: mostly yes, but with a sharp warning for late transformer layers.
What Muon actually does (and where it can break)
While Adam treats each parameter as an independent scalar, Muon recognizes that neural network weights are matrices with geometric structure and optimizes them accordingly. Concretely, Muon makes each weight matrix update orthonormal through polar decomposition via Newton-Schulz iterations, giving direction-only, spectrally controlled steps for hidden layers. The goal is to produce an update where every singular value equals 1 -- meaning every direction in weight space gets updated with equal magnitude, no direction dominates, and no direction gets ignored.
The catch is that computing the exact polar decomposition requires a full SVD at every training step, which is prohibitively expensive. In large-scale optimization, Newton-Schulz (NS) iterations provide fast, inexact polar decomposition updates as cheap alternatives to SVD-based projection. The key recurrence is run for a small, fixed number of steps -- typically 5 -- balancing approximation error and computational budget. But NS is only an approximation. Directions whose singular values are too small fail to be properly orthonormalized. If a momentum matrix has very small singular values, the 5-step NS pass simply does not push them close enough to 1 -- those directions get undertreated, and the update quality degrades silently.