Skip to main content
Back

Meta's FAIR Cuts AI Reasoning Training Costs 40% by Reusing Discarded Data

A FAIR/Meta paper shows that reusing old rollouts in a replay buffer cuts LLM RL training compute by up to 40% — without hurting accuracy, and sometimes improving it.

Pro

  • 40% compute savings: A well-designed replay buffer cuts LLM RL training compute by up to 40% while matching or beating on-policy accuracy.
  • Challenges core assumption: The paper proves mathematically that strict on-policy sampling is suboptimal when rollout generation is expensive (which it always is — inference costs 80%+ of RL GPU hours).
  • Three-way trade-off formalized: The optimal buffer design balances staleness, sample diversity, and the relative cost of inference vs. training — with a closed-form solution provided.
  • Bonus: training stability: Replay buffers prevent training crashes and preserve output diversity (better pass@k), effects not predicted by theory alone.
  • Minimal code change: Replacing a LIFO queue with a circular buffer that doesn't delete on sample is all that's needed; pseudocode is in the paper appendix.
  • Tested on multiple models: Results confirmed on Qwen3-0.6B, Qwen2.5-7B, Qwen3-8B (coding), and Llama 3.2 3B; frontier-scale validation still needed.

Every time you train a reasoning model with RL, the most expensive part is not the gradient updates , it is generating the rollouts. In state-of-the-art pipelines, inference cost often dominates the training budget, consuming more than 80% of post-training GPU hours. And yet, the standard practice is to throw those rollouts away after a single use. A new paper from FAIR at Meta and NYU argues that this "generate-then-discard" habit is simply wasteful, and they have the math and experiments to prove it.

Experience Replay , the practice of storing rollouts and reusing them multiple times during training , is a foundational technique in general RL, yet it remains largely unexplored in LLM post-training due to the prevailing belief that fresh, on-policy data is essential for high performance. This paper challenges that assumption. The result: experience replay can cut LLM RL training compute by up to ~40%.

Why everyone assumed stale data was poison

Modern LLM RL pipelines , think PPO or GRPO used to train reasoning models , are designed to be as "on-policy" as possible. That means the model generates a batch of responses, uses them for exactly one gradient update, and discards them. This stands in stark contrast to classical RL, where experience replay , storing and reusing past trajectories in a buffer , is a foundational tool for sample efficiency.

The fear is "off-policiness" (also called staleness): if you train on data generated by an older version of your policy, the gradient signal is biased. The model has moved on, but the training data has not. In robotics and gaming RL, this is managed carefully. In LLM training, the community largely decided it was not worth the trouble , until now.

The three-way trade-off the paper formalizes

The paper formalizes the optimal buffer design as a three-way trade-off between staleness-induced noise growth, coupling between replayed samples and the parameter iterates, and the rollout-vs-training compute imbalance. The key insight is that when rollout generation is expensive (which it always is for LLMs), the math shifts decisively in favor of reuse.