Tsinghua Reveals Why Stronger Teachers Fail in On-Policy Distillation
A Tsinghua paper dissects why on-policy distillation fails, revealing two hard rules and two practical fixes that every LLM post-training pipeline should know
Takeaways
- Two hard rules: OPD only works when teacher and student share compatible thinking patterns AND the teacher brings genuinely new knowledge beyond the student's training.
- Benchmark score is misleading: A 7B teacher from the same family as a 1.5B student caused identical regression as a weaker 1.5B teacher, despite scoring higher on benchmarks.
- Token-level mechanism: Successful OPD is driven by progressive alignment on shared high-probability tokens (97-99% of probability mass); failing runs show stagnant overlap from the start.
- Two fixes for failing OPD: Off-policy cold-start SFT before OPD, and using prompts from the teacher's own post-training data, both reliably rescue failing configurations.
- Dense reward has a ceiling: Reward quality degrades with trajectory depth; instability starts at later tokens and propagates backward, limiting OPD for long-horizon reasoning.
- Code released: Full implementation available at github.com/thunlp/OPD from Tsinghua's THUNLP group.
On-policy distillation (OPD) has quietly become one of the most powerful tools in LLM post-training. Recent industry efforts, including Qwen3, MiMo, and GLM-5, all adopt OPD in their post-training pipelines and report substantial gains, establishing it as a competitive complement to conventional supervised fine-tuning and outcome-reward reinforcement learning. The core idea is elegant: instead of training a student model on fixed teacher-generated sequences, you have the student generate its own outputs and use the teacher's per-token probability scores as a dense reward signal. The student learns from its own mistakes, on states it actually visits.
But OPD has a dirty secret. Despite these successes, OPD remains poorly understood and fragile in practice, with a striking failure mode where a stronger teacher can completely fail to improve a student, even when a weaker teacher succeeds from lower initial alignment. A new paper from Tsinghua University, "Rethinking On-Policy Distillation of Large Language Models", is the most systematic attempt yet to explain why.
Two rules that govern everything
The paper identifies two necessary conditions for OPD to work. Violate either one and your training will stall or regress, regardless of how strong the teacher is.
- Thinking-pattern consistency. The student and teacher must share compatible reasoning styles. A teacher that reasons differently from the student produces a token-level signal the student simply cannot exploit, even if that teacher scores higher on benchmarks.
- New knowledge, not just scale. The teacher must bring genuinely new capabilities the student has not already seen during its own training. A larger model from the same training pipeline often fails this test, because it has converged to a similar distribution at a different scale.
The authors validate these findings through weak-to-strong reverse distillation, showing that same-family 1.5B and 7B teachers are distributionally indistinguishable from the student's perspective. Probing into the token-level mechanism, they show that successful OPD is characterized by progressive alignment on high-probability tokens at student-visited states, a small shared token set that concentrates most of the probability mass (97%-99%).