One Hidden Ratio Controls How Fast AI Models Actually Learn
A new study shows that learning rate and weight norm act on training loss almost entirely through their ratio, unifying how weight decay, Hyperball, and schedules shape pretraining.
PRO- New paper shows learning rate and weight norm act on loss dynamics almost entirely through their ratio, the effective learning rate
- When ELR is matched across runs, loss curves collapse with errors of a few x 10^-3, below seed noise
- Holds across optimizers, architectures, datasets, model scales, and norm-control methods
- Weight decay and Hyperball shape loss dynamics mainly through the ELR schedules they induce
- Swapping raw LR for ELR lets fitted scaling laws transfer across norm-control recipes without refit
- ELR framing explains delayed acceleration, the crossover where norm-controlled runs catch up late in training
Pretraining a language model is a game of tuning knobs that all seem to interact with each other. Learning rate, weight decay, normalization design, and now optimizer wrappers like Hyperball each pull on the loss curve in ways that are hard to disentangle. A new paper argues that most of this apparent complexity is an illusion, because a single quantity, the effective learning rate, quietly does almost all of the work.
The authors of Effective Learning Rate Governs Loss Dynamics uncover what they call ELR collapse: when the ratio of learning rate to parameter norm is matched across runs, the loss trajectories line up on top of each other, even when the raw learning rates and weight norms are wildly different.
The knobs that would not stop tangling
Modern transformer training is full of scale-invariant subnetworks. Because layers like LayerNorm and RMSNorm make the loss insensitive to the magnitude of upstream weights, the true speed of learning is not the raw learning rate but the learning rate divided by the parameter norm. This idea has been floating around for years, with prior work noting that a normalization layer causes the subnetwork preceding it to become scale invariant, so the layer's effective learning rate depends on the norm of its parameters, and when the norm grows the effective learning rate shrinks.
Every popular norm-control trick, including decoupled weight decay, Hyperball, and the specific placement of LayerNorm blocks, changes that parameter norm in its own way. That makes cross-comparisons between recipes messy. A learning rate schedule that looks aggressive in one setup can be gentle in another, because the denominator quietly shifted.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
