Moonshot AI's Moonlight-16B Cuts LLM Training Costs in Half by Ditching AdamW
Moonshot AI's Moonlight proves the Muon optimizer scales to 16B-parameter MoE models, cutting training compute in half while beating Llama and Qwen at the same scale.
PRO- Moonshot AI releases Moonlight-16B-A3B-Instruct, a 16B MoE model (3B active params) trained with the Muon optimizer instead of AdamW.
- Muon achieves ~2x compute efficiency over AdamW, requiring only ~52% of training FLOPs for equivalent performance.
- Moonlight outperforms Llama3.2-3B (trained on 9T tokens) and DeepSeek-V2-Lite on nearly every benchmark, using only 5.7T tokens.
- Two key fixes enabled Muon to scale: adding weight decay and normalizing update magnitudes across matrix shapes.
- Critical caveat: Muon fine-tuning on Adam-pretrained models does not improve over Adam; gains are only captured at pretraining time.
- Model weights, code, and intermediate checkpoints are open-sourced under MIT license; compatible with vLLM and SGLang.
The optimizer you use to train a large language model has always been a settled question: AdamW, full stop. Moonshot AI's Moonlight-16B-A3B-Instruct challenges that assumption head-on. It is a 16B-parameter Mixture-of-Experts model trained entirely with the Muon optimizer, and the results make a strong case that AdamW's reign as the default training algorithm for LLMs may be coming to an end.
What is Muon, and why does it matter?
Muon stands for MomentUm Orthogonalized by Newton-Schulz. The core idea is simple but powerful: instead of treating each weight as an independent scalar (as Adam does), Muon treats weight matrices as geometric objects and orthogonalizes the gradient momentum before applying the update. Orthogonalization here means rotating the gradient update so that it spreads learning signal evenly across all directions of the weight matrix, rather than over-optimizing a few dominant directions.
While Adam treats each parameter as an independent scalar, Muon recognizes that neural network weights are matrices with geometric structure and optimizes them accordingly. This turns out to matter enormously at scale. It orthogonalizes the momentum matrix before each update, achieving approximately 2x compute efficiency over Adam while requiring less memory by eliminating the second moment.
Muon had already shown promise on small models, but nobody had proven it could work at the scale of a production LLM trained on trillions of tokens. That is exactly what Moonlight demonstrates.
The two fixes that made Muon scale
The Moonshot team identified two critical failure modes that prevented vanilla Muon from scaling, and fixed both:
- Weight Decay: Without weight decay, model weights and layer outputs grew unboundedly during long training runs, eventually exceeding the precision range of bfloat16. Adding standard AdamW-style weight decay resolved this and improved final performance beyond both vanilla Muon and AdamW.
- Consistent RMS Updates: Muon's update magnitude varies depending on the shape of each weight matrix. For large MLP matrices, updates were too small; for small attention head matrices, updates were too large. The fix: scale each matrix's update by the square root of its largest dimension, normalizing the update magnitude across all parameter types. This eliminated training instabilities and removed the need for per-layer hyperparameter tuning.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
