NVIDIA's Lightning OPD Trains Qwen3-8B 4x Faster Without a Live Teacher

On-policy distillation has quietly become one of the most effective ways to squeeze reasoning capability into smaller language models, but it comes with an ugly bill: you have to keep a giant teacher model running on GPUs the entire time you're training the student. A new paper from NVIDIA argues that most of that infrastructure is unnecessary, provided you obey one condition that almost every published OPD pipeline currently violates.

The framework, called Lightning OPD, precomputes the teacher's per-token log-probabilities once, ahead of training, then trains the student against that fixed cache. Starting from an SFT-initialized Qwen3-8B-Base model, it reaches 69.9% on AIME 2024 in just 30 GPU hours, a 4.0x speedup over standard OPD, while matching accuracy on every benchmark the authors evaluated.

Why the teacher server was the bottleneck

On-policy distillation (OPD) sits between supervised fine-tuning and reinforcement learning. Rather than a sparse pass or fail reward, the student generates its own rollouts and the teacher scores every token, giving a dense per-token advantage signal of the form log pi_T(a_t) - log pi_theta(a_t). This dense token-level KL signal is what makes OPD compute-efficient compared to RL from verifiable rewards, which only provides a sparse trajectory-level reward that scales poorly when most rollouts fail the verifier.

The catch is that the teacher has to be online. Every gradient step, the student produces new rollouts, and a multi-GPU teacher server has to score them on the fly. For an 8B student with a 32B teacher, you end up co-hosting both models, fragmenting your cluster and making experiments painful to reproduce outside of a well-funded lab.

The obvious workaround is to precompute the teacher's log-probabilities once over some fixed rollouts and reuse them. When the authors tried this naively, it broke: the offline variant failed to match standard OPD, and they set out to find out why.