Peking University's ICPO Beats GRPO by 4.1 Points Without a Teacher Model
A new reinforcement learning recipe uses in-context demonstrations from existing datasets to expand exploration, beating GRPO by up to 4.1 points on math benchmarks.
Takeaways
- ICPO replaces expensive expert-model distillation with in-context demonstrations from existing datasets to steer RL exploration.
- Beats GRPO by up to +4.1 average points on math benchmarks (AIME, AMC, MATH-500, Minerva, Olympiad).
- Surpasses LUFFY by +3.3 and Prefix-RFT by +1.6 points on Qwen2.5-Math-7B without needing a stronger teacher model.
- Three components: implicit expert forcing, expert region reject sampling, and annealed expert-bonus reward shaping.
- Works even on base models with no ICL capability, expanding Pass@32 substantially over GRPO.
- Code and training recipe available at github.com/Celine-hxy/ICPO, built on VERL.
Reinforcement learning with verifiable rewards has become the standard recipe for pushing reasoning models past supervised fine-tuning. The popular Group Relative Policy Optimization (GRPO) algorithm has a well-known flaw though: it only samples from the model's current policy, so exploration is bounded by whatever the model already knows how to do. A team from Peking University and Tencent has proposed a fix called In-Context Steered Policy Optimization (ICPO), and it works without needing a bigger teacher model.
The core idea is disarmingly simple. Instead of paying to distill trajectories from an advanced Large Reasoning Model, ICPO exploits the model's own in-context learning ability to generate expert-conditioned rollouts using demonstrations pulled from existing datasets. Those steered rollouts get mixed into GRPO training as off-policy samples, nudging the policy toward regions of the solution space it would rarely visit on its own.

Why on-policy sampling hits a wall
To understand the problem, remember what GRPO does. For each prompt, it samples a group of rollouts from the current policy, computes rewards, and normalizes advantages within the group. There is no critic model, which makes it cheap. Because every rollout comes from the current policy's distribution, though, trajectory diversity is narrow and training often converges to local optima.
Recent work tried to break out of this trap by mixing in supervised fine-tuning traces from stronger models. Approaches like LUFFY, ReLIFT, and Prefix-RFT feed expert trajectories into the RL loop as off-policy samples. That helps, but generating extra reasoning traces from advanced models is expensive and those models are not always accessible. External trajectories are also noisy, and dumping them straight into training can destabilize gradients.