CMU Collapses Four AI Training Stages Into One Shared Loss
A CMU-led team collapses pretraining, SFT, and RL into one maximum-likelihood objective, letting you mix static text and RL rollouts in a single trainer.
- CMU-led team unifies pretraining, SFT, and RL into one maximum-likelihood objective called unified training.
- RL rollouts reframed as likelihood fitting to model-generated targets via MaxRL extension.
- Single trainer mixes static text and RL generations; staged training becomes a special case.
- Beats staged training on accuracy, compute efficiency, coverage, and forgetting in controlled reasoning experiments.
- Produces better starting checkpoints for continual learning across math then coding RL.
- Code and checkpoints released at GitHub and Hugging Face.
Large language model training commonly moves through four pipelines: pretraining on broad corpora, midtraining on curated data, supervised fine-tuning (SFT), and reinforcement learning (RL) from model-generated rollouts. Each stage brings its own data, objective, scheduler, and infrastructure. A new paper from Carnegie Mellon University and collaborators derives a likelihood-based algorithm that can train across all four stages.
A shared objective turns the allocation of tokens, examples, and rollouts into a configurable data-mixing problem. Developers can vary when each source enters training, how much compute it receives, and how long it remains in the mix without changing optimization algorithms between stages. The experiments report stronger reasoning performance, less post-training forgetting, and better preparation for later RL tasks under matched compute budgets.
Why the Pipeline Split
| Stage | Typical input | Typical objective |
|---|---|---|
| Pretraining | Broad text and code corpora | Next-token likelihood |
| Midtraining | Curated domain data | Next-token likelihood |
| SFT | Prompt-response examples | Response likelihood |
| RL | Model rollouts scored by rewards | Policy gradient |
Pretraining, midtraining, and SFT already optimize closely related cross-entropy losses. Their main differences lie in data selection, masking, and learning-rate schedules. RL methods such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO) instead generate responses, score them with a reward function, and estimate policy gradients from those scores.
That optimization split usually produces separate trainers, rollout services, replay systems, and optimizer configurations. It also complicates pretraining-data replay during RL, leaves compute allocation to manual experiments, and makes scaling-law analysis difficult across the boundary between pretraining and post-training. Models can also lose general capabilities when later updates concentrate on a narrow reward.
One Loss, Many Data Sources
Maximum Likelihood RL (MaxRL), an earlier result from the same group, expresses a policy-gradient update as maximum-likelihood fitting to a success-conditioned distribution derived from the current policy. In plain terms, rewarded generations become weighted training targets that a cross-entropy trainer can consume.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
