Kakao Trains a 155B MoE on 10T Tokens Without a Single Hyperparameter Sweep

Training a Mixture-of-Experts model at frontier scale involves a painful chicken-and-egg problem. The learning rate you pick determines whether the run converges or explodes, but sweeping learning rates on a 100B-plus parameter model burning through trillions of tokens is out of the question. A team at Kakao Corp has released a paper showing you can skip almost all of that guesswork by decomposing the problem into two clean, cheap steps, then extrapolating.

They put the recipe to the test by pretraining a 155B total / 17B active MoE foundation model from scratch on 10 trillion tokens, with a stable loss curve and benchmark scores landing on the Pareto frontier next to dots.llm1 and GLM-4.5-Air.

Why the usual playbook breaks at MoE scale

The go-to method for avoiding hyperparameter sweeps in dense models is Maximal Update Parameterization, or muP. It scales neural network parameters to maintain O(1) updates across layers, ensuring stable feature learning as model width increases, which means a good learning rate found on a tiny model transfers directly to a much wider one.

MoE architectures throw a wrench in that. They add sparsity as a second scaling axis, governed by how many experts fire per token relative to the total pool. The sparsity patterns and routing mechanisms intrinsic to MoE architectures fall outside the scope of current muP theory, and simply widening the hidden dimension becomes impractical at 100B-plus parameters because inference costs balloon. The practical move is to grow the expert count while keeping active experts fixed, but nobody had established whether muP's transfer guarantees survive that maneuver.

Even if width transfer works, the optimal learning rate also shifts as the token budget grows. Sweeping across both model size and token count simultaneously is the exact 2D grid the authors want to avoid.