Skip to main content
Back

UCLA and Princeton's SDPG Trains Reasoning Models Faster Without Collapse

SDPG unifies verifier-based RL and self-distillation into a single training objective, fixing the sparse-reward and mode-collapse problems that plague both approaches separately

  • SDPG combines GRPO-style verifier rewards with full-vocabulary on-policy self-distillation in a single principled loss, fixing sparse rewards and mode collapse simultaneously.
  • A key theoretical result shows that reverse-KL self-distillation is locally equivalent to a policy gradient step with a centered log teacher/student ratio as the token advantage.
  • Two stabilizers prevent the distillation signal from destabilizing training: positive advantage gating (only distill on correct rollouts) and a warmup-decay beta schedule.
  • On Qwen3-4B, SDPG-UFKL reaches 0.408 on AIME24 and 0.870 on AMC23, outperforming GRPO (0.316 / 0.739) and RLSD (0.395 / 0.813) across all benchmarks.
  • SDPG avoids the entropy collapse seen in pure self-distillation (RLSD entropy hits zero by step 250), maintaining healthy exploration throughout training.
  • Code is open-sourced on top of the verl framework, with scripts for Qwen3-4B and Qwen3-1.7B experiments and all four KL regularization modes.

Reinforcement learning from verifiable rewards (RLVR) has become the go-to recipe for post-training reasoning models. Algorithms like GRPO sample groups of responses, score them with a rule-based verifier, and push the policy toward correct answers. The problem is that the reward signal is brutally sparse: a single binary score is spread uniformly across every token in a potentially thousand-token response. The model has no idea which tokens in a correct chain of reasoning actually mattered.

A parallel line of work tries to fix this with self-distillation. The idea is elegant: show the model the correct answer as a hint, run a forward pass to get a "privileged" token distribution, then train the model without the hint to match that distribution. Dense, per-token supervision for free. The catch is that pure self-distillation has no verifier, so it can happily reinforce locally plausible tokens on globally wrong trajectories, and it tends to collapse into mode-seeking behavior over time.

Researchers from UCLA and Princeton now propose SDPG (Self-Distilled Policy Gradient), a framework that combines both sources of supervision in a principled way, while adding two stabilizers that prevent each component from undermining the other.

The core insight: self-distillation is policy gradient in disguise

The key theoretical contribution is a gradient identity. For a fixed rollout prefix and with the teacher branch detached, the student-side gradient of the reverse-KL full-vocabulary OPD loss equals a detached-sampling policy-gradient update whose token advantage is a centered log teacher/student ratio. In plain terms: minimizing the KL between the model-without-hint and the model-with-hint is mathematically equivalent to running a policy gradient step where each token's advantage is proportional to how much the hint boosted its probability. This is a gradient identity, not an approximation, and it means the two training signals can be cleanly combined in a single loss.