Skip to main content
Back

Michigan Researchers Beat the Data Wall With MIR, Gaining 1.3x More Training Data

A new scaling law and masking trick show how to squeeze more out of repeated training data as the internet runs dry

Paper, Full Access
  • Data wall problem: Chinchilla scaling law assumes single-pass training on unique data, breaking down as models train for multiple epochs on finite corpora.
  • MIR regularization: Masked-Input Regularization adds a masked-context next-token prediction loss at training time, borrowing from diffusion LMs with zero inference overhead.
  • Best combo: MIR on top of strong weight decay outperforms either alone; gains are measurable at 1.4B parameters on downstream benchmarks.
  • SoftQ scaling law: Proposed as a replacement for Chinchilla in the multi-epoch regime; uses multiplicative coupling of model and data size instead of additive decoupling, achieving R² > 0.999 on experimental fits.
  • Data-efficiency estimate: SoftQ estimates MIR's benefit as equivalent to ~1.3x more unique training data.
  • Open source: Code released at github.com/yixinw-lab/dc_pretrain; paper at arxiv.org/abs/2606.06888.

The internet is not infinite. As frontier model training runs consume trillions of tokens, the field is quietly entering a regime where the data runs out before the compute budget does. The Chinchilla scaling law established robust empirical trends in performance as a joint function of model size and dataset size under the one-pass training paradigm, in which each data point is used at most once. That assumption is becoming increasingly untenable. A new paper from researchers at the University of Michigan attacks this problem head-on, proposing both a better regularization strategy and a more accurate scaling law for the world where models train on the same data over and over again.

The data wall is already here

Training compute is increasingly outpacing the availability of high-quality data, shifting the central challenge from optimal compute allocation to extracting maximum value from limited data. The simplest response is repeating data across multiple epochs, and it is already widespread. But repetition has a cost: the model starts to memorize rather than generalize, and validation loss eventually climbs back up.

The scaling laws guiding modern model training were calibrated for a single regime: data-rich, single-epoch pretraining. The dominant Chinchilla form has three structural limitations outside that regime: it diverges as unique data shrinks instead of saturating at the uninformed baseline; it cannot represent overfitting when capacity exceeds the data; and it conflates total examples seen with unique examples available. In other words, the map is wrong for the territory we are now in.

Two problems, two solutions

The paper attacks the data-constrained regime along two independent axes: how to regularize a model that keeps seeing the same text, and how to accurately predict loss when the standard scaling law formula breaks down.

On the regularization side, the authors introduce Masked-Input Regularization (MIR). The idea is simple: during training, randomly mask out some fraction of the input tokens, then ask the model to still predict the next token correctly. This is an auxiliary loss stacked on top of the standard autoregressive objective. It borrows the core trick from diffusion language models (dLLMs) -- which train by learning to fill in randomly corrupted text -- and grafts it onto a standard causal transformer. The masked-diffusion objective regularizes the model and prevents it from overfitting, while the autoregressive objective provides rapid initial learning.