CalibAdv Fixes the Training Collapse Killing GRPO-Based Search Agents
CalibAdv fixes two chronic failures in GRPO training for search agents: unfair penalties on correct intermediate steps and catastrophic language collapse.
Takeaways
- CalibAdv fixes GRPO for search agents by calibrating advantages using intermediate step correctness. Paper
- 44% of penalized intermediate steps in standard GRPO on Qwen2.5-7B are actually correct.
- A silver document proxy scores each step by overlap with documents retrieved by successful rollouts.
- Final-step advantage rebalancing with lambda=1.0 eliminates catastrophic language collapse entirely.
- Average F1 improves 11.8% over Search-R1 across three models and seven QA benchmarks.
- Code is available on GitHub, no annotations or extra models needed.
Training a language model to become a competent search agent, one that reasons, queries a retriever, reads results, and iterates until it lands on the right answer, has become the poster child use case for Group Relative Policy Optimization (GRPO). The recipe is simple: sample a group of full rollouts, score them by whether the final answer is correct, and push the policy toward the winners while pushing it away from the losers. That simple recipe breaks in two ugly ways in practice, and a new paper called CalibAdv offers a targeted fix for both.
Two failure modes hiding inside GRPO
The first problem is credit assignment. GRPO takes the reward from the final answer and broadcasts the same advantage to every token in the rollout. If the final answer is wrong, every intermediate <search> query and <think> block in that trajectory gets a negative advantage, even when the model actually issued a perfectly reasonable query that retrieved the correct document. The authors quantify this: on Qwen2.5-7B, roughly 44% of penalized intermediate steps are likely correct and being punished by accident.
The second problem is training collapse. Past a certain step, models trained with vanilla GRPO on multi-turn search start producing garbled Unicode, then degenerate into word-level repetition dominated by the <think> token. Diagnosing this, the authors track output entropy, sampled-token probabilities, perplexity, and the ratio of negative to positive advantages. The picture is consistent: negative advantages persistently dominate positive ones, entropy first spikes then collapses, PPL explodes, and the policy loses coherent language.