Skip to main content
Back

Yale Explains Why Muon Beats Adam by Navigating 1.76x Flatter Terrain

A new paper finally explains why Muon trains LLMs 2x faster than Adam -- the answer lies in how each optimizer navigates the curvature of the loss landscape

Paper, Full Access
  • 2x efficiency explained: A new paper shows Muon outperforms Adam in LLM training because it steps in directions with lower curvature (NDS), not because it takes bigger steps.
  • Key metric -- NDS: Normalized Directional Sharpness measures how sharp the loss surface is along the optimizer's update direction; Muon's NDS is ~1.76x lower than Adam's on average.
  • Data imbalance amplifies the gap: On Zipf-distributed (realistic language) data, Muon's NDS advantage grows 1.8x wider compared to uniform data -- real corpora make Muon even better.
  • Orthogonalization balances curvature: Theoretically proven that Muon spreads update energy evenly across high- and low-curvature directions, while Adam concentrates energy in sharp directions.
  • Within-layer effect dominates: In mid-to-late training, Muon's advantage comes from smaller within-layer curvature, not cross-layer interactions.
  • No new code or model: This is a theoretical analysis paper; the full paper is on arXiv with no accompanying implementation.

Ever since the Muon optimizer burst onto the scene and demonstrated roughly a 2x training efficiency gain over Adam for large language models, the ML community has been asking the same question: why does it work so well? The original blog post from Keller Jordan famously attributed Muon's success to "divine benevolence." A new paper from researchers at Yale takes a more rigorous approach, offering the first geometric explanation for Muon's edge -- and the answer comes down to curvature.

The Problem With "It Just Works"

Despite Muon's empirical success, a rigorous understanding of why and how it outperforms Adam in transformers has remained incomplete. That's a real problem. When you don't understand why an optimizer works, you can't predict when it will fail, how to tune it, or how to improve it. Muon consistently outperforms Adam on various LLM training tasks ranging from small-scale benchmarks to larger settings, and is more data-efficient than Adam, requiring fewer tokens to reach the same validation loss. But the mechanism behind that efficiency has been a black box.

What Muon Actually Does

To understand the paper's findings, it helps to know what makes Muon different from Adam in the first place. While Adam treats each parameter as an independent scalar, Muon recognizes that neural network weights are matrices with geometric structure -- and optimizes them accordingly.

Concretely, Muon is a matrix orthogonalization-based optimizer specially designed for two-dimensional parameters of hidden layers in a neural network. The key step is orthogonalization: Muon regularizes the gradients with respect to weight matrices by orthogonalization, employing a few Newton-Schulz iterations on the matrix momentum to normalize all singular values to 1. Think of it as reshaping the gradient update so that no single direction dominates -- every direction of change gets equal weight.

In practice, Muon is typically applied only to 2D weight matrices in hidden layers, while scalar parameters, bias vectors, embeddings, and unembedding layers continue to use standard optimizers such as Adam. So it's not a wholesale replacement -- it's a targeted intervention on the parts of the network where matrix structure matters most.