Self-play pretraining turns programs into a corpus

Researchers have demonstrated a pretraining system that uses no human-generated training corpus. In a new paper titled Self-Play Pretraining with Zero Data, two randomly initialized networks bootstrap each other by generating programs and learning to predict the byte sequences those programs produce.

In the paper, “zero data” refers to the absence of an external dataset. Training sequences still exist, but they come entirely from programs sampled during training. On unseen datasets containing text, images, audio, and melodies, the learner’s zero-shot validation loss falls predictably as self-play compute increases. It also develops in-context learning and discovers recognizable mathematical sequences.

Escaping a finite corpus

Large language models consume growing quantities of text, while high-quality human material remains finite. Synthetic-data pipelines can extend that supply, although they usually rely on teacher models already trained on curated human content. The new method removes that dependency from its pretraining loop.

Solomonoff induction provides the theoretical motivation. This uncomputable prediction framework assigns higher probability to sequences generated by shorter programs. Such programs capture reusable structures including repetition, composition, and recursion, which also appear throughout natural data. The researchers test whether models can acquire those structures through computation and transfer them to unfamiliar modalities.

Programs become the curriculum

Both networks begin with random weights and improve together through a repeated generation, execution, and optimization loop: