A 7.4B Model Matches GPT-3 13B Using 20x Less Compute

New research shows architectural tricks like recursive depth and model growth can bend pretraining scaling exponents, delivering compute savings that compound.

·
·
·
A 7.4B Model Matches GPT-3 13B Using 20x Less ComputePRO
  • New paper argues architectural choices can change pretraining scaling exponents, not just constants.
  • A 7.4B model growth architecture matches GPT-3 13B on CORE using 20x less compute.
  • Reported compute multipliers grow with scale: 1.55x at 1e20 FLOPs, projected 2.7x at 1e25.
  • Three interventions studied: looping, model growth (tied and untied), and a boundary operator that normalizes residuals.
  • In data-constrained multi-epoch training, looping acts as a regularizer worth about 2.2x compute efficiency.
  • Code released at qlabs-eng/scaling-exponents with growth scripts and boundary operator.

Empirical scaling laws usually describe how language-model loss falls as training compute, parameter count, and data increase. Chinchilla-style laws also estimate the compute-optimal balance between model size and training tokens. A paper by Zixi Chen, Akshay Vegesna, Samip Dahal, and Andrew Gordon Wilson reports that three architectural changes alter the fitted scaling exponent itself: recursive looping, model growth, and a boundary operator.

The reported gains widen with scale. An untied-growth model with 7.4 billion parameters matched GPT-3 13B on the CORE evaluation suite while using about one-twentieth of the pretraining compute. The result applies to that benchmark and compute accounting, but it challenges the common assumption that architecture can deliver only fixed, constant-factor savings.

A steeper exponent compounds

Compute scaling laws often take the form L(C) = B + AC−α, where L is loss, C is training compute, B is an estimated irreducible floor, and α controls the curve’s slope. Lowering A shifts the curve by a fixed amount. Increasing α makes loss fall faster, so the equivalent-compute advantage grows as training runs become larger.

The authors measure that advantage as a compute multiplier: the amount of compute a standard transformer would need to reach the same loss. Their best measured variant reaches about 1.55× at 1020 FLOPs, with the fitted law projecting 2.7× at 1025 FLOPs.

Depth that still changes predictions

The proposed mechanism centers on computational depth, defined as the number of layers that meaningfully change the model’s prediction. Pre-normalized transformers can lose useful depth because the residual stream grows while each later block contributes a progressively smaller update. The network remains deep in layer count, yet its upper layers exert less influence on the output.

Pro article

This story is for Pro members

You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.

Trending
  • No trending articles

Comments

avatar

Next Reads