Skip to main content
Back

PGPO Borrows Fluid Physics to Stop Self-Teaching AI From Collapsing

PGPO borrows viscous-fluid physics to dynamically scale gradient steps in self-distillation training, fixing the instability that causes SDPO to collapse

  • PGPO fixes a core instability in Self-Distilled Policy Optimization (SDPO), where fixed step sizes cause training collapse late in runs.
  • The fix uses a mutual-information estimate between student and teacher outputs to scale each gradient step -- big steps when the teacher is informative, small steps when it is not.
  • The approach is inspired by viscous-fluid dynamics, formalized as a stochastic differential equation (SDE) modification to standard SGD.
  • On ScienceQA, PGPO outperforms SDPO on 3 of 4 domains with gains up to +4.5 accuracy points, and stays stable where SDPO collapses.
  • The method adds negligible compute overhead per iteration and preserves the convergence guarantees of vanilla SGD.
  • No code or model weights have been released; results are currently limited to the ScienceQA benchmark.

Post-training large language models with reinforcement learning is hard enough. Post-training them using only their own outputs as a teacher -- no external model, no separate reward network -- is harder still. That is exactly what Self-Distilled Policy Optimization (SDPO) attempts, and it has been gaining serious traction. But SDPO carries a subtle flaw that can quietly wreck a training run, and a new paper proposes an unusual fix: borrow the math of viscous fluids.

The self-distillation wave

Large language models are increasingly post-trained with reinforcement learning in verifiable domains such as code and math, yet current methods for reinforcement learning with verifiable rewards (RLVR) learn only from a scalar outcome reward per attempt, creating a severe credit-assignment bottleneck. SDPO was designed to break that bottleneck. SDPO converts tokenized feedback into a dense learning signal without any external teacher or explicit reward model, treating the current model conditioned on feedback as a self-teacher and distilling its feedback-informed next-token predictions back into the policy -- leveraging the model's ability to retrospectively identify its own mistakes in-context.

In practice, this means the same model plays two roles simultaneously: a student that generates answers, and a self-teacher that sees the graded feedback and rewrites those answers token-by-token. The student then learns to imitate the self-teacher's revised outputs. No separate model is needed, which makes SDPO cheaper and simpler than distillation from a larger external teacher.

The crack in the foundation

SDPO is sensitive to how much each update step should be trusted: corrections from a self-teacher can be highly informative on some batches and misleading on others, and applying them uniformly with a fixed step size can destabilize training. This is not a minor edge case. The self-teacher's quality fluctuates batch to batch -- sometimes it produces genuinely corrective rewrites, and sometimes it hallucinates confident nonsense. A fixed learning rate treats both cases identically, and late in training that can cause the policy to collapse entirely.