Teaching Small AI Models to Learn From Their Own Mistakes

Knowledge distillation lets smaller language models inherit capabilities from larger systems, but the standard recipe creates a train-test gap. During training, the student reads polished teacher completions. During inference, it must continue from its own imperfect tokens, allowing early mistakes to distort everything that follows. Mingyang Song and Mao Zheng’s new survey, A Survey of On-Policy Distillation for Large Language Models, organizes research that addresses this gap by training students on their own generated trajectories.

As a literature review, the paper contributes a shared formalism, taxonomy, and research agenda. It reports no new benchmark results and ships no code.

Perfect prefixes make brittle students

Standard distillation is off-policy: the student learns from a fixed corpus of teacher-generated sequences. Its loss is calculated on histories written by the teacher, even though deployment exposes it to histories written by the student. Once the model produces an unlikely token, it enters a context that may be poorly represented in its training data.

Theoretical analyses of imitation learning show that cumulative error can grow quadratically with sequence length under common assumptions. Long reasoning traces, tool-using agents, and multi-step workflows amplify the problem because each action affects the context for every later decision.

On-policy distillation, or OPD, changes which histories receive supervision. The student samples a rollout, and the teacher evaluates student-generated prefixes through token distributions, sequence scores, or outcome rewards. Repeating this sampling and correction loop exposes the student to states it is likely to encounter during deployment. Under interactive imitation-learning assumptions, that process can reduce error growth toward a linear dependence on trajectory length.