CMU and Microsoft's MOPD Teaches AI to Learn From Its Own Failures
MOPD teaches LLMs by showing them their own mistakes alongside successes, yielding consistent gains over GRPO and SDPO across code, math, science, and tool-use tasks
PRO- MOPD is a new on-policy distillation framework that conditions the teacher on both successful and failed peer rollouts from the same prompt, producing sharper token-level supervision.
- The optimal peer context is 2 successes + 1 failure, achieving pairwise accuracy of 0.47 vs. 0.13 for no-peer baseline in teacher signal quality analysis.
- On LiveCodeBench v6, MOPD reaches 61.82 mean@8 with Qwen3-8B, vs. 49.49 for SDPO and 43.65 for GRPO.
- MOPD also beats SDPO on math (AIME2024: 28.54 vs. 7.29 mean@8), science QA (all 4 domains), and tool use benchmarks.
- Peer conditioning produces nearly 2x more diverse outputs than SDPO (Distinct-1: 0.052 vs. 0.027) and improves structural code diversity by 17%.
- Code is available on GitHub; the method adds only 9–33% compute overhead over SDPO and works in both self-distillation and teacher-student settings.
When a language model is being trained to reason, it typically generates several attempts at each problem and gets a simple thumbs-up or thumbs-down from a verifier. That sparse signal tells the model whether it succeeded, but nothing about where its reasoning went wrong. A new paper from Carnegie Mellon University and Microsoft proposes a surprisingly intuitive fix: show the model its own successful and failed attempts side by side, and use that contrast to build a sharper training signal.
The method, called Multi-Rollout On-Policy Distillation (MOPD), is a peer-conditioned distillation framework. It consistently outperforms standard on-policy baselines across competitive programming, mathematical reasoning, scientific QA, and tool-use benchmarks, with code available on GitHub.
The training signal problem
Post-training LLMs for reasoning usually involves one of two approaches. Reinforcement learning with verifiable rewards (RLVR) , the paradigm behind DeepSeek-R1 and similar models , uses sparse binary rewards: did the model get the answer right or wrong? This creates a fundamental credit-assignment bottleneck: a single scalar reward per attempt indicates whether the model was wrong, but not where or why.
On-policy distillation (OPD) is a complementary approach that provides denser, token-level supervision. OPD has emerged as a paradigm that provides dense, token-level supervision from a teacher model, achieving strong performance on reasoning-intensive tasks. The key advantage of on-policy training is that the student is supervised on trajectories it actually generates, avoiding the distribution mismatch that plagues offline methods where a model is trained on states it never visits at inference time.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
