Hugging Face's FinePhrase Beats Every Synthetic Dataset Using 24x Less Compute
Hugging Face ran 90 controlled experiments to figure out what actually makes rephrased web text useful for pretraining, and released a 486B-token dataset built on the findings.
Takeaways
- Hugging Face released FinePhrase, a 486B-token synthetic pretraining dataset with an open pipeline.
- Structured pedagogical prompts (math, faq, table, tutorial) beat all prior synthetic rephrasing methods.
- Generator model scale saturates near 1B parameters; 27B models perform worse than 1B models.
- SmolLM2 1.7B is the best generator because its output diversity avoids template collapse.
- FinePhrase-Table hits 17.18 macro-avg versus 13.77 for DCLM, a 3.41-point gain.
- Generation cost is roughly 30x lower than REWIRE's Llama 3.3 70B pipeline; code is on GitHub.
Synthetic pretraining data has quietly become one of the most important levers in modern LLM development, yet almost every recipe published so far compares itself to a different baseline with a different generator model and different source text. Hugging Face just tried to fix that with a controlled ablation across the three axes that matter, and the resulting corpus, FinePhrase, ships as 486 billion tokens of rephrased web text under an open license.
The team ran 90 experiments, generated over 1 trillion tokens, and spent 12.7 GPU years searching for the best recipe. FinePhrase outperforms all existing synthetic data baselines, and the paper is unusually blunt about which design choices matter and which do not.
Why rephrasing web text became a thing
Pretraining data has moved in three waves: curated corpora like Wikipedia and BookCorpus, then scaled web crawls like C4 and The Pile, and finally trillion-token filtered crawls like FineWeb and DCLM. As the crawlable web plateaus, teams have started using LLMs to rephrase existing documents into new variants. Nemotron-CC rephrased two trillion tokens, and several recent LLMs used hundreds of billions of synthetic tokens during pretraining.
Comparability is the problem. WRAP focuses on stylistic rewriting, Nemotron-CC extracts QA pairs and knowledge lists, BeyondWeb proposes continuation and summarization rephrasing, and REWIRE targets guided transformation of low-quality documents. No one had compared these under matched conditions, and every paper claimed a win against its own baseline.
The setup
Each configuration is a tuple of a rephrasing prompt, an instruction-tuned generator model, a source dataset (what gets rephrased), and a mix-in dataset (original web text blended into training). The team validates each configuration by training a 1.2B Qwen 2 architecture language model from scratch on a mixture of rephrased samples and mix-in data for 21B tokens. Evaluation spans 12 benchmarks including ARC, MMLU Redux, SQuAD v2, DROP, GSM8K, HellaSwag, and WikiTableQuestions.