PGPO Borrows Fluid Physics to Stop Self-Teaching AI From Collapsing
PGPO borrows viscous-fluid physics to dynamically scale gradient steps in self-distillation training, fixing the instability that causes SDPO to collapse
- PGPO fixes a core instability in Self-Distilled Policy Optimization (SDPO), where fixed step sizes cause training collapse late in runs.
- The fix uses a mutual-information estimate between student and teacher outputs to scale each gradient step -- big steps when the teacher is informative, small steps when it is not.
- The approach is inspired by viscous-fluid dynamics, formalized as a stochastic differential equation (SDE) modification to standard SGD.
- On ScienceQA, PGPO outperforms SDPO on 3 of 4 domains with gains up to +4.5 accuracy points, and stays stable where SDPO collapses.
- The method adds negligible compute overhead per iteration and preserves the convergence guarantees of vanilla SGD.
- No code or model weights have been released; results are currently limited to the ScienceQA benchmark.
Post-training large language models with reinforcement learning is hard enough. Post-training them using only their own outputs as a teacher -- no external model, no separate reward network -- is harder still. That is exactly what Self-Distilled Policy Optimization (SDPO) attempts, and it has been gaining serious traction. But SDPO carries a subtle flaw that can quietly wreck a training run, and a new paper proposes an unusual fix: borrow the math of viscous fluids.
The self-distillation wave
Large language models are increasingly post-trained with reinforcement learning in verifiable domains such as code and math, yet current methods for reinforcement learning with verifiable rewards (RLVR) learn only from a scalar outcome reward per attempt, creating a severe credit-assignment bottleneck. SDPO was designed to break that bottleneck. SDPO converts tokenized feedback into a dense learning signal without any external teacher or explicit reward model, treating the current model conditioned on feedback as a self-teacher and distilling its feedback-informed next-token predictions back into the policy -- leveraging the model's ability to retrospectively identify its own mistakes in-context.
In practice, this means the same model plays two roles simultaneously: a student that generates answers, and a self-teacher that sees the graded feedback and rewrites those answers token-by-token. The student then learns to imitate the self-teacher's revised outputs. No separate model is needed, which makes SDPO cheaper and simpler than distillation from a larger external teacher.
The crack in the foundation
SDPO is sensitive to how much each update step should be trusted: corrections from a self-teacher can be highly informative on some batches and misleading on others, and applying them uniformly with a fixed step size can destabilize training. This is not a minor edge case. The self-teacher's quality fluctuates batch to batch -- sometimes it produces genuinely corrective rewrites, and sometimes it hallucinates confident nonsense. A fixed learning rate treats both cases identically, and late in training that can cause the policy to collapse entirely.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
