Skip to main content
Back

Harvard and Stanford's PST Lets Small AI Models Train Each Other Without Labels

A new label-free fine-tuning method uses cross-model agreement as its own training signal, improving math reasoning by up to 4.3 points.

  • Peer-Predictive Self-Training (PST) lets multiple small LLMs improve on math reasoning without any labels or reward models.
  • Models generate answers sequentially; the last response acts as an aggregated reference signal for training.
  • Each intermediate response is scored by pointwise mutual information, then used to weight self-training updates.
  • Accuracy improved 2.2-4.3 points across Gemma-2-2B, LLaMA-3.2-1B, and Qwen2.5-1.5B on SimulEq, MATH-500, and MultiArith.
  • PST beat self-training, majority-vote SFT, and GRPO baselines while cutting the generator-verifier gap by 26-40%.
  • Full training code and the paper are publicly available.

A team from Harvard and Stanford has proposed a way for a group of small language models to train each other on reasoning tasks without any labels, reward models, or teacher-student setup. Their method, called Peer-Predictive Self-Training (PST), improves math accuracy on three benchmarks by 2.2 to 4.3 percentage points across Gemma-2-2B, LLaMA-3.2-1B, and Qwen2.5-1.5B, using nothing but the models talking to each other.

The core trick treats a chain of sequential model responses as an implicit verifier. Rather than asking one model to grade another, PST measures how much each model's answer helped predict the group's aggregated final answer, then uses that as a training weight.

The label problem that won't go away

Continued self-improvement of language models without external supervision remains an open problem. Most successful post-training pipelines rely on labeled data, explicit reward models, or engineered feedback signals, which limits scalability when ground truth is expensive, incomplete, or unavailable. Classical self-training loops that fine-tune a model on its own predictions tend to reinforce their own mistakes, a well-known confirmation bias problem.

PST leans on a computational asymmetry that shows up across recent reasoning research: models often struggle to generate correct answers, yet retain the ability to recognize or validate correct reasoning when shown candidate solutions. If verification is easier than generation, then multiple weak generators voting together can produce a signal stronger than any of them alone. That's the wisdom-of-crowds intuition, formalized as a training loop.