---
title: "One Hidden Ratio Controls How Fast AI Models Actually Learn"
subtitle: "A new study shows that learning rate and weight norm act on training loss almost entirely through their ratio, unifying how weight decay, Hyperball, and schedules shape pretraining."
url: "https://alphasignal.ai/news/one-hidden-ratio-controls-how-fast-ai-models-actually-learn"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-08-25T16:57:29.000Z"
updated: "2026-08-27T00:04:54.406Z"
type: "Paper"
topics:
  - "Training Infra"
  - "Llms"
subtopics:
  - "Scaling Laws"
  - "Pretraining"
upvotes: 14
source_url: "https://arxiv.org/abs/2608.24814"
comments: 0
truncated: true
---

# One Hidden Ratio Controls How Fast AI Models Actually Learn

> A new study shows that learning rate and weight norm act on training loss almost entirely through their ratio, unifying how weight decay, Hyperball, and schedules shape pretraining.

Pretraining a language model is a game of tuning knobs that all seem to interact with each other. Learning rate, weight decay, normalization design, and now optimizer wrappers like Hyperball each pull on the loss curve in ways that are hard to disentangle. A new paper argues that most of this apparent complexity is an illusion, because a single quantity, the effective learning rate, quietly does almost all of the work.

The authors of [Effective Learning Rate Governs Loss Dynamics](https://arxiv.org/abs/2608.24814) uncover what they call ELR collapse: when the ratio of learning rate to parameter norm is matched across runs, the loss trajectories line up on top of each other, even when the raw learning rates and weight norms are wildly different.

## The knobs that would not stop tangling

Modern transformer training is full of scale-invariant subnetworks. Because layers like LayerNorm and RMSNorm make the loss insensitive to the magnitude of upstream weights, the true speed of learning is not the raw learning rate but the learning rate divided by the parameter norm. This idea has been floating around for years, with [prior work](https://arxiv.org/html/2407.01800v1) noting that a normalization layer causes the subnetwork preceding it to become scale invariant, so the layer's effective learning rate depends on the norm of its parameters, and when the norm grows the effective learning rate shrinks.

Every popular norm-control trick, including decoupled weight decay, Hyperball, and the specific placement of LayerNorm blocks, changes that parameter norm in its own way. That makes cross-comparisons between recipes messy. A learning rate schedule that looks aggressive in one setup can be gentle in another, because the denominator quietly shifted.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining](https://arxiv.org/abs/2608.24814)
- [Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization](https://arxiv.org/abs/2606.16899)
- [Efficient Hyperparameter Tuning via Trajectory Invariance Principle](https://arxiv.org/html/2509.25049v1)
- [Normalization and effective learning rates in reinforcement learning](https://arxiv.org/html/2407.01800v1)
- [Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks](https://arxiv.org/html/2507.02119v1)