Perplexity and NVIDIA's JOLT Beats GRPO Using 6.5x Fewer Training Tokens
Perplexity and NVIDIA introduce JOLT, a self-distillation post-training method that matches GRPO accuracy on MATH using 6.5x fewer training tokens.
- Perplexity and NVIDIA released JOLT, a joint teacher-student self-distillation post-training method.
- On MATH, JOLT matches GRPO accuracy with 6.5x fewer training tokens and up to 14x fewer completion tokens.
- JOLT+ beats GRPO on all five benchmarks tested, including +9.9pp on AppWorld and +4.2pp on MATH.
- One model plays both roles: teacher sees privileged info (answers, traces), student does not; shared weights updated jointly.
- Teacher is trained to give feedback aligned with student's current ability, not just to maximize its own task reward.
- Full paper available on arXiv, tested on Qwen3 and DeepSeek-V2-Lite base models.
Perplexity Research and NVIDIA have introduced JOLT, or Joint On-policy Learning and Teaching, a post-training method in which one model serves as both teacher and student. The teacher receives privileged training information and learns to provide guidance the student can follow. On MATH, JOLT matches GRPO accuracy with an estimated 6.5 times fewer training tokens. Its enhanced variant, JOLT+, outperforms GRPO across the reported math, coding, tool-use, and terminal-agent benchmarks.
Why sparse rewards waste rollouts
Group Relative Policy Optimization, or GRPO, generates several attempts for a prompt, compares their outcome rewards, and increases the probability of actions associated with stronger results. A single pass/fail score attached to a long trajectory provides little information about which intermediate steps helped or hurt. That sparse feedback limits sample efficiency on agentic tasks, where trajectories are long and successful attempts are rare.
On-policy distillation adds a denser signal by having a teacher score each token generated by the current student policy. A high-performing solver can still provide unusable instruction when its guidance assumes access to answers, execution traces, or reasoning steps unavailable to the student. The student then receives locally plausible targets that exceed its current capabilities.
One model, two views
JOLT extends OPSD, or On-Policy Self-Distillation. A shared set of model weights serves both roles, with each role receiving different context. The student sees the task as it will appear during deployment. The teacher also sees privileged information such as verified answers, worked solutions, execution traces, or verifier hints.
- Generate on-policy attempts. The student samples trajectories from its current policy, so training follows the behavior the model currently produces.
- Rescore every token. The shared model processes each trajectory again with privileged context and produces teacher probabilities for the student’s generated tokens.
- Update the student. A distillation objective moves the student toward those dense token-level probabilities.
- Update the teacher. The teacher optimizes task reward with Kullback-Leibler regularization, which measures the distance between its action distribution and the student’s. A larger regularization coefficient keeps the guidance closer to the student’s current behavior.
- Add direct reward in JOLT+. JOLT+ combines the distillation objective with the student’s own outcome-reward objective.
The JOLT paper defines a reward-consistent teacher as one whose distillation gradient aligns with the policy gradient of the student’s expected reward. In practical terms, the teacher’s token-level guidance should increase the student’s probability of completing the task successfully.
Inputs, passes, and deployment
- Privileged training context: reference solutions, verifier feedback, successful traces, or other information correlated with task success.
- An outcome signal: a reward or verifier used to optimize the teacher and, with JOLT+, the student.
- Additional training passes: student trajectories must be rescored with the teacher context.
- A tuned KL coefficient: this controls how far the teacher can move beyond the student’s current distribution.
- Standard deployment input: inference uses the student view, so privileged training data is unnecessary after post-training.
Gains across five benchmarks
The evaluation covers five benchmarks and several Qwen3 checkpoints. JOLT+ records the highest mean score in every reported setting, with the following comparisons against GRPO.
| Benchmark | Task | Reported JOLT+ result |
|---|---|---|
| GSM8K | Grade-school math | Accuracy higher by 1.5 percentage points |
| MATH | Competition math | Accuracy higher by 4.2 percentage points |
| LiveCodeBench v6 | Code generation | Accuracy higher by 3.0 percentage points |
| AppWorld | Tool use | State-test pass rate higher by 9.9 percentage points |
| Terminal-Bench 2.1 | Terminal agent | Best reported pass@k at every tested k, from 10.96% at pass@1 to 34.83% at pass@32 |
The experiments use models including Qwen3-0.6B on MATH, Qwen3-8B on LiveCodeBench v6, Qwen3-4B-Instruct on AppWorld, and Qwen3-30B-A3B-Instruct on Terminal-Bench. Pass@k measures whether at least one of k sampled attempts succeeds, so the Terminal-Bench figures show performance as the sampling budget increases.
Vanilla JOLT, which omits the student’s direct outcome-reward objective, still beats both distillation baselines. It trails GRPO on AppWorld and Terminal-Bench, indicating that direct task feedback remains useful for long-horizon agent tasks.
Token savings depend on the accounting
At a matched MATH accuracy of 63.44%, the observed GRPO checkpoint consumes 8.40 million tokens. JOLT’s fitted scaling curve estimates 1.30 million tokens, a 6.5-fold reduction. Because the comparison pairs an observed checkpoint with a fitted curve, the ratio is an estimate rather than a direct checkpoint-to-checkpoint measurement.
Completion-only accounting produces larger ratios because it excludes prompts and privileged context. Across target accuracies from 55% to 65%, JOLT uses 9.5 to 14.4 times fewer completion tokens than GRPO. Under the broader processed-token measure, which includes prompts and privileged context, the reduction ranges from 4.3 to 7.4 times.
The MATH scaling curves also reveal different training profiles. JOLT improves quickly and levels off near 72% accuracy within the reported budget. OPSD rises quickly at small budgets and plateaus around 62%. GRPO starts more slowly and continues improving to roughly 70%.
A fit for verifier-rich training
JOLT is best suited to tasks where training already provides reliable privileged information, including reference solutions, executable tests, environment state, or verifier hints. It converts those assets into token-level supervision while using a single shared model for the teacher and student roles.
The shared weights remove the need to host a separate, larger teacher model. Training still requires teacher-context rescoring and additional token processing, so the reported token efficiency does not establish an equivalent reduction in wall-clock time or total floating-point operations. Hardware utilization, sequence lengths, and the cost of constructing privileged context will affect the operational gain.
The KL anchor addresses guidance that assumes capabilities the student has not acquired. Keeping the teacher distribution near the student’s current policy makes its targets more reachable, while the reward objective preserves pressure toward successful behavior. The method’s benefit therefore depends on the quality of the verifier, the privileged context, and the chosen regularization strength.
The authors also propose applying joint training to a task-generating teacher and a problem-solving student, allowing synthetic curricula to adapt as the student improves. For current pipelines, the reported results favor JOLT+ when direct rewards are available and long-horizon success remains sparse.
Comments
