KAUST's Gluon Cuts Distributed AI Training Bandwidth by 65% Without Losing Speed
Four new compressed optimizers extend Muon-style training to federated settings, cutting uplink traffic by roughly 60-65% while preserving convergence.
ProTakeaways
- Four new compressed variants of the Gluon optimizer for distributed Muon-style LLM training.
- Uses SARAH-style variance reduction on gradient differences to shrink compression error to zero.
- Communication cost drops from O(nd/ε³) to O(√n·d/ε²) versus prior EF21-Muon.
- Experiments show 60% bandwidth savings on logreg, 65% on CIFAR-10 CNN.
- Yields new byproducts: faster uncompressed algorithm, Local Gluon, Local Gluon with error-compensated MVR.
- Supports both unbiased (RandK) and contraction (TopK) compressors with error feedback.
Muon has quickly become the optimizer to beat for large language model pretraining, but it has a scaling problem that Adam did not: spread it across hundreds of nodes and the network becomes the bottleneck. A new paper from KAUST titled Communication-Efficient Gluon in Federated Learning proposes four compressed variants of the Gluon optimizer that shrink the amount of data workers exchange, while preserving the convergence guarantees that made Muon-style methods interesting in the first place.
Why Muon broke the network
Muon outperformed AdamW empirically when training a 1.5B-parameter NanoGPT model, and that efficiency advantage held up on Mixture-of-Experts language models with up to 16B parameters. The catch is that Muon-type optimizers use a linear minimization oracle (LMO) over non-Euclidean norm balls, meaning they compute update directions by solving a small constrained problem in a matrix norm like the spectral norm, rather than doing plain gradient descent. That works well on one machine, but at scale communication cost dominates.
Gluon, the layer-wise generalization of Muon that this paper builds on, updates the parameters layer-wise instead of jointly over the full vector X. The distinction matters because for extremely high-dimensional models, computing a single global LMO for the entire parameter vector is prohibitively expensive, while breaking the problem into per-layer LMOs keeps things tractable. Gluon also uses a layer-wise