Harvard and Microsoft Research Expose How RL Training Secretly Destroys AI Coverage
A new theoretical analysis shows REINFORCE-based methods like PPO and GRPO can silently kill the base model's best responses, and offers a fix.
- New theory paper identifies coverage collapse as a core failure mode in PPO and GRPO post-training.
- REINFORCE-based methods can push optimal-response probability to near zero while reward still looks healthy.
- KL regularization to a fixed base model does not prevent the collapse, only slows it.
- Fix is Approximate Policy Mirror Descent: periodically reset the KL reference to the current policy.
- Authors prove first global convergence guarantees for coverage-preserving PPO, GRPO, and Kimi-style squared-loss variants.
- Guarantees hold for softmax linear policies over pretrained features, with sample complexity scaling with base-model coverage.
Researchers at the University of Illinois Urbana-Champaign, Harvard, Princeton, and Microsoft Research have formalized a failure mode in reinforcement learning post-training called coverage collapse. Their analysis shows that REINFORCE-based methods, including stylized versions of PPO and GRPO, can suppress useful responses until optimization requires exponentially many steps. Periodically moving the KL reference policy to the current policy can prevent that collapse under the paper’s assumptions.
The 141-page preprint, titled A Theory of Policy Optimization in Post-Training: Coverage Collapse and Reference Resets, is authored by Audrey Huang, Nan Jiang, Sham M. Kakade, Sadhika Malladi, Akshay Krishnamurthy, and Dylan J. Foster. It investigates why RL post-training can stall even when average reward continues to improve.
A rising reward can hide lost coverage
Post-training commonly assumes that a pretrained model already assigns some probability to high-quality responses. RL then increases that probability. The paper studies response coverage, roughly the probability that the base policy samples responses required by an optimal policy. Low initial coverage means the learner needs more samples to encounter those responses.
The authors call an algorithm coverage-preserving when its sample and gradient-step requirements scale with the base model’s initial response coverage. This property captures the intended role of post-training: amplify behavior already available through sampling.
In the paper’s constructed instances, REINFORCE updates shift probability toward an early high-reward response and suppress alternatives needed to reach the global optimum. Reward improves during this shift, so the reward curve can conceal the loss of coverage. Once the relevant responses become sufficiently unlikely, their gradients become tiny and progress stalls. The same behavior occurs with exact gradients, which separates the failure from sampling noise. A fixed-reference KL penalty can delay the collapse while leaving the exponential worst case intact.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
