Fudan and Baidu's AAPO Fixes the Silent Flaw Stalling GRPO-Trained Reasoning Models
A new reinforcement learning method fixes the zero-gradient problem plaguing GRPO by adding a reward margin against a frozen reference model.
Takeaways
- AAPO adds an advantage margin computed against a frozen reference model to fix GRPO's zero-gradient problem in late training.
- The paper shows zero-advantage proportion hitting 100% in late GRPO training, wasting compute on null updates.
- AAPO-7B scores 52.4 average on five math benchmarks, beating GPG-7B by 10.3% and SimpleRL-Zero-7B by 12.2%.
- Method optimizes cross-entropy loss weighted by group advantage plus clipped policy-vs-reference reward margin.
- Cost is 28-46% more wall-clock per step from extra reference model sampling, no additional GPU memory.
- Code is public on GitHub with training scripts for Llama and Qwen model families.
Reinforcement learning has become the dominant recipe for teaching large language models to reason, but the workhorse algorithm behind DeepSeek-R1 and its imitators has a quiet efficiency problem. A team from Fudan University and Baidu's Frontier Research Department has published AAPO (Advantage-Augmented Policy Optimization), a drop-in replacement for GRPO that keeps training signals alive even when the policy gets stuck in the flat regions where GRPO effectively stops learning.
The paper reports consistent gains across Llama and Qwen model families on five math reasoning benchmarks, with the 7B variant hitting a 52.4 average on the standard suite. Code is available on GitHub.
The zero-advantage trap
Group Relative Policy Optimization (GRPO), the algorithm DeepSeek popularized, replaces PPO's value network with a cheaper trick: sample a group of responses to the same prompt, score them, and use each response's deviation from the group mean as its advantage. It works, but it has a failure mode that gets worse as training progresses.
When a group's rewards are all similar, whether uniformly correct or uniformly wrong, the advantage approaches zero. Low variance within a group produces zero gradient and no parameter updates. The authors' statistical analysis shows this is common rather than an edge case: the zero-advantage proportion approached 100% in later training steps with increasing frequency.
The reason is intuitive. Once a policy learns the easy patterns, its samples for a given prompt cluster in quality. All responses look roughly the same to the reward model, group standard deviation collapses, and the normalized advantage goes to zero. The gradient goes with it. You are burning GPU-hours to compute updates of magnitude approximately zero.