---
title: "NVIDIA's Lightning OPD Trains Qwen3-8B 4x Faster Without a Live Teacher"
subtitle: "NVIDIA researchers show that on-policy distillation can drop its live teacher server, cutting 8B reasoning training from 120 to 30 GPU hours."
url: "https://alphasignal.ai/news/nvidia-s-lightning-opd-trains-qwen3-8b-4x-faster-without-a-live-teacher"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "NVIDIA AI"
company_url: "https://alphasignal.ai/company/nvidiaai"
published: "2026-04-14T00:00:00.000Z"
updated: "2026-08-23T23:27:44.561Z"
type: "Paper"
topics:
  - "Post Training"
  - "Reasoning"
subtopics:
  - "Distillation"
  - "Math Reasoning"
  - "Test Time Compute"
upvotes: 1190
source_url: "https://arxiv.org/abs/2604.13010"
comments: 0
truncated: true
---

# NVIDIA's Lightning OPD Trains Qwen3-8B 4x Faster Without a Live Teacher

> NVIDIA researchers show that on-policy distillation can drop its live teacher server, cutting 8B reasoning training from 120 to 30 GPU hours.

On-policy distillation has quietly become one of the most effective ways to squeeze reasoning capability into smaller language models, but it comes with an ugly bill: you have to keep a giant teacher model running on GPUs the entire time you're training the student. A new paper from NVIDIA argues that most of that infrastructure is unnecessary, provided you obey one condition that almost every published OPD pipeline currently violates.

The framework, called [Lightning OPD](https://github.com/jet-ai-projects/Lightning-OPD), precomputes the teacher's per-token log-probabilities once, ahead of training, then trains the student against that fixed cache. Starting from an SFT-initialized Qwen3-8B-Base model, it reaches 69.9% on AIME 2024 in just 30 GPU hours, a 4.0x speedup over standard OPD, while matching accuracy on every benchmark the authors evaluated.

## Why the teacher server was the bottleneck

On-policy distillation (OPD) sits between supervised fine-tuning and reinforcement learning. Rather than a sparse pass or fail reward, the student generates its own rollouts and the teacher scores every token, giving a dense per-token advantage signal of the form `log pi_T(a_t) - log pi_theta(a_t)`. This dense token-level KL signal is what makes OPD compute-efficient compared to RL from verifiable rewards, which only provides a sparse trajectory-level reward that scales poorly when most rollouts fail the verifier.

The catch is that the teacher has to be online. Every gradient step, the student produces new rollouts, and a multi-GPU teacher server has to score them on the fly. For an 8B student with a 32B teacher, you end up co-hosting both models, fragmenting your cluster and making experiments painful to reproduce outside of a well-funded lab.

The obvious workaround is to precompute the teacher's log-probabilities once over some fixed rollouts and reuse them. When the authors tried this naively, it broke: the offline variant failed to match standard OPD, and they set out to find out why.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation](https://arxiv.org/abs/2604.13010)
- [Lightning OPD GitHub Repository](https://github.com/jet-ai-projects/Lightning-OPD)
- [On-Policy Distillation LLMs Redefine Post-Training Efficiency](https://www.startuphub.ai/ai-news/ai-research/2025/on-policy-distillation-llms-redefine-post-training-efficiency)
- [Awesome On-Policy Distillation](https://github.com/chrisliu298/awesome-on-policy-distillation)