AI Web Text Changes the Economics of Pretraining

In an August 2026 web crawl, researchers from Pangram Labs and the University of Maryland classified 31% of tokens as AI-generated. A new study then trained 800 language models on controlled mixtures of human and AI web text to measure how each source affects next-token prediction loss. The result depends on the training regime: generated tokens can help data-starved models, but their marginal value saturates and eventually becomes negative as human-data budgets and model sizes grow. Training plans based on the Chinchilla scaling law miss that reversal because they assign equal value to every token.

The team’s estimated AI share rose from roughly 10% in June 2024 to 27.5% in June 2026 and 31% two months later. Common quality filters increased that concentration. FineWeb’s pipeline retained AI-labeled documents at 2.3 times the rate of human-labeled documents, while DCLM’s retained them at 9.8 times the rate. A filtered crawl can therefore contain a higher proportion of generated text than its source crawl.

Equal-token math reaches its limit

Earlier synthetic-data research largely examined purpose-built training corpora or recursive model-collapse experiments in which models learn from their own outputs. Wild web text has a broader origin. It comes from systems such as GPT-4o, Claude, and Llama, may include human edits, and appears alongside human writing across many topics and formats. The WildAI experiments isolate this material in controlled training mixtures.