Smaller batches improve pretraining on repeated data

Language-model teams often reuse a fixed, curated corpus for several epochs, meaning the model processes each token more than once. Sewon Min’s preprint finds that repeated-data training changes the optimum: smaller global batches, fresh sequence packing, tied embeddings, and embedding weight decay can improve held-out loss. The study also shows that GPU count and wall-clock time place separate constraints on the best configuration.

Common pretraining recipes are tuned for runs that encounter mostly unique tokens. Teams working with domain-specific, licensed, filtered, or otherwise limited datasets face a different problem because repetition increases the risk of memorization and overfitting. Settings that maximize hardware throughput can therefore produce a worse model from the same corpus.

A fixed corpus changes the optimum

Min calls the first experimental setting infinite-compute pretraining. The term means the optimizer may take as many steps as needed over a fixed pool of unique tokens, allowing the experiments to isolate data repetition from hardware limits. Wall-clock and GPU constraints return in the paper’s later analysis.

Batch size and epoch count are tuned together because they determine how often the model sees each token and how many optimizer updates it receives. Fixing one while sweeping the other can hide the best configuration, especially when a small batch creates many more sequential updates per epoch.

The loss curve bends back up

Validation loss follows a U-shaped curve as global batch size increases. Loss initially improves, reaches a minimum, and then worsens at batch sizes below those commonly used in large distributed training jobs. The precise optimum varies with model size, dataset size, and training duration, so the paper supports a batch sweep rather than a universal target.