Fudan's CW-GRPO Teaches AI to Pinpoint Which Web Searches Actually Matter
A new twist on GRPO uses an LLM judge to score each search round and redistribute credit toward the moves that actually solved the task.
Takeaways
- CW-GRPO adds per-round contribution scoring to GRPO for training LLM search agents (paper).
- An LLM judge scores each round on retrieval utility and reasoning correctness, both binary.
- Scores rescale outcome advantages across the trajectory instead of being optimized as rewards.
- Reports 5.0% gain on Qwen3-8B and 6.3% on Qwen3-1.7B versus standard GRPO.
- Code released on GitHub; uses GPT-oss-120B as judge, veRL framework.
- Only redistributes credit for successful trajectories; failed ones get uniform weighting.
Training a language model to run its own web searches is deceptively hard. The model fires off queries, reads snippets, reasons over them, then answers, and the only signal you get back is whether the final answer was right. A team from Fudan University has proposed Contribution-Weighted GRPO (CW-GRPO), a reinforcement learning method that figures out which individual search rounds inside a successful trajectory actually did the work, then pushes learning signal toward those specific moves.
Reported gains are modest but consistent: a 5.0% relative improvement over standard GRPO on Qwen3-8B and 6.3% on Qwen3-1.7B across seven knowledge-intensive question-answering benchmarks. Code is available on GitHub.
The credit assignment headache
Existing RL approaches for search agents split into two camps, each with a well-known failure mode. Process supervision assigns round-level rewards and is commonly optimized with actor-critic methods like PPO, but learning reliable critics over diverse intermediate states is unstable and frequently produces biased advantages and brittle training. Outcome supervision only checks the final answer, which is stable to train but treats every search round in a winning trajectory as equally important, even though most trajectories contain one or two pivotal queries surrounded by filler.
Group Relative Policy Optimization, which powers methods like Search-R1, improves training stability and memory efficiency without altering the sparsity of the reward signal, leaving the credit assignment issue unresolved. That is the gap CW-GRPO tries to close without giving up GRPO's stability.