Skip to main content
Back

Princeton's SD-Zero Teaches AI to Fix Its Own Mistakes Without a Teacher

Princeton researchers train models to grade their own homework, turning binary correct/incorrect signals into dense token-level supervision without external teachers.

Pro

  • SD-Zero trains one model as both generator and reviser, turning binary rewards into dense token-level supervision.
  • No external teacher or gold solutions needed, only a correctness checker and the model itself.
  • Gains of 10.5% on Qwen3-4B and 10.4% on Olmo-3-7B across eight math and code benchmarks.
  • Beats GRPO, SDFT, and RFT under matched sampling budgets while cutting response length by roughly half.
  • Reviser produces highly localized token feedback, functioning like a free process reward model.
  • Code available at princeton-pli/Self-Distillation-Zero; paper on arXiv.

Reinforcement learning with verifiable rewards (RLVR) has become the default recipe for teaching language models to reason, but it has a well-known efficiency problem: a single right-or-wrong signal per response tells the model nothing about which of its thousands of tokens actually caused the failure. A new method called Self-Distillation Zero (SD-Zero) from Princeton, CMU, and Toronto proposes a fix that needs no external teacher and no gold-standard solutions, just the model itself and a binary correctness check.

SD-Zero trains a single model to play two roles: a Generator that produces an initial response, and a Reviser that conditions on that response and its binary reward to produce an improved one. It then performs on-policy self-distillation to fold the reviser back into the generator, using the reviser's token distributions as supervision. The model learns to convert a scalar 0/1 signal into per-token feedback by asking itself what it would do differently if told the answer was wrong.

Comparison of RLVR sparse supervision, SD-Zero dense conversion, and on-policy self-distillation approaches

The sparse-versus-dense trade-off

Post-training in verifiable domains has generally split into two camps. RLVR methods like GRPO are broadly applicable because they only need a checker, but they force the model to figure out good reasoning by contrasting many self-generated rollouts. Distillation methods give rich token-level targets but need either a stronger external teacher or high-quality demonstrations. More recent self-distillation approaches, including OPSD, SDFT, and SDPO, drop the external teacher yet still lean on demonstrations that must be much better than what the model produces on its own.