---
title: "PGPO Borrows Fluid Physics to Stop Self-Teaching AI From Collapsing"
subtitle: "PGPO borrows viscous-fluid physics to dynamically scale gradient steps in self-distillation training, fixing the instability that causes SDPO to collapse"
url: "https://alphasignal.ai/news/pgpo-borrows-fluid-physics-to-stop-self-teaching-ai-from-collapsing"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-06-02T00:00:00.000Z"
updated: "2026-06-10T06:46:35.617Z"
type: "Paper"
topics:
  - "Post Training"
  - "Reasoning"
subtopics:
  - "Distillation"
  - "Rl"
  - "Math Reasoning"
upvotes: 11
source_url: "https://arxiv.org/abs/2606.03620"
comments: 0
truncated: true
---

# PGPO Borrows Fluid Physics to Stop Self-Teaching AI From Collapsing

> PGPO borrows viscous-fluid physics to dynamically scale gradient steps in self-distillation training, fixing the instability that causes SDPO to collapse

Post-training large language models with reinforcement learning is hard enough. Post-training them using only their own outputs as a teacher -- no external model, no separate reward network -- is harder still. That is exactly what **Self-Distilled Policy Optimization (SDPO)** attempts, and it has been gaining serious traction. But SDPO carries a subtle flaw that can quietly wreck a training run, and a new paper proposes an unusual fix: borrow the math of viscous fluids.

## The self-distillation wave

Large language models are increasingly post-trained with reinforcement learning in verifiable domains such as code and math, yet current methods for reinforcement learning with verifiable rewards (RLVR) learn only from a scalar outcome reward per attempt, creating a severe credit-assignment bottleneck. SDPO was designed to break that bottleneck. SDPO converts tokenized feedback into a dense learning signal without any external teacher or explicit reward model, treating the current model conditioned on feedback as a self-teacher and distilling its feedback-informed next-token predictions back into the policy -- leveraging the model's ability to retrospectively identify its own mistakes in-context.

In practice, this means the same model plays two roles simultaneously: a **student** that generates answers, and a **self-teacher** that sees the graded feedback and rewrites those answers token-by-token. The student then learns to imitate the self-teacher's revised outputs. No separate model is needed, which makes SDPO cheaper and simpler than distillation from a larger external teacher.

## The crack in the foundation

SDPO is sensitive to how much each update step should be trusted: corrections from a self-teacher can be highly informative on some batches and misleading on others, and applying them uniformly with a fixed step size can destabilize training. This is not a minor edge case. The self-teacher's quality fluctuates batch to batch -- sometimes it produces genuinely corrective rewrites, and sometimes it hallucinates confident nonsense. A fixed learning rate treats both cases identically, and late in training that can cause the policy to collapse entirely.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Physics-Guided Policy Optimization with Self-Distillation (arXiv)](https://arxiv.org/abs/2606.03620)
- [Reinforcement Learning via Self-Distillation (SDPO paper)](https://arxiv.org/abs/2601.20802)
- [SDPO Official Website](https://self-distillation.github.io/SDPO)
- [Awesome On-Policy Distillation (GitHub)](https://github.com/chrisliu298/awesome-on-policy-distillation)
- [ScienceQA Benchmark Leaderboard](https://llm-stats.com/benchmarks/scienceqa)