Peking University's RL-PLUS Fixes the Hidden Collapse Killing Math AI
A new hybrid-policy training method fixes the pass@k regression seen in RLVR-trained reasoning models and pushes math scores past strong baselines.
Takeaways
- RL-PLUS fixes the pass@k collapse where RLVR-trained models fall below their own base model at high k.
- Combines on-policy GRPO with an off-policy term consuming external correct reasoning trajectories.
- Multiple Importance Sampling uses a mixture denominator to bound variance and control bias when integrating external data.
- Focal-loss-style advantage weighting
(1-p)^gammaamplifies gradients for correct-but-unlikely tokens, suppressing already-mastered ones. - Hits 53.4 average on six math benchmarks with Qwen2.5-Math-7B, beating SFT+GRPO by 5.2 and LUFFY by 3.3.
- Code available on GitHub, paper accepted at ACL 2026.
Reinforcement learning with verifiable rewards has become the go-to recipe for making language models better at math and code. Reward the model when its answer matches ground truth, penalize it when it doesn't, and it gets sharper at pass@1. A nagging problem hides underneath the leaderboard numbers though: the trained model often becomes worse than its own base when you let it try many times. A new paper from Peking University and Alibaba's Tongyi Lab, RL-PLUS, tackles this and was accepted to ACL 2026.
The collapse hiding behind pass@1 gains
RLVR has advanced complex reasoning in LLMs, but it struggles to break through the inherent capability boundaries of the base model. The essentially on-policy strategy, combined with an immense action space and sparse rewards, leads to capability boundary collapse that narrows the model's problem-solving scope. In practice this shows up as an ugly pattern in pass@k curves: an RLVR-trained model beats the base at pass@1, ties around pass@16, and loses at pass@128. The RL step sharpens a subset of behaviors the base already had while quietly killing off the long tail of reasoning paths it could sometimes stumble into.
The authors trace this to two coupled issues. On-policy RL only rewards trajectories the current policy actually generates, so with sparse binary rewards on long chains of thought, the model has almost no gradient signal for genuinely new reasoning patterns. Meanwhile, entropy collapses over training, so exploration keeps shrinking. Adding a supervised fine-tuning loss on top of RL, a common workaround, often makes things worse in their tests, dropping average math accuracy from 45.5 to 40.1.