LoopRPT Teaches AI to Think Smarter With Every Recurrent Pass
A new reinforcement pre-training recipe pushes rewards into the hidden iterations of looped transformers, improving accuracy and cutting compute per token.
- LoopRPT is a reinforcement pre-training framework built for looped language models like Ouro.
- It reframes next-token prediction as a per-iteration reasoning task with rewards assigned to latent steps.
- Uses an EMA teacher reference and noisy latent rollouts to supervise intermediate hidden states.
- Instantiated on Ouro-1.4B and 2.6B, it achieves Pareto dominance on accuracy versus compute trade-offs.
- Gains concentrate on hard MATH500 tokens, suggesting better early reasoning rather than premature exits.
- Full paper on arXiv; code not yet released, Ouro backbone is open.
LoopRPT trains language models inside the latent loop
Looped language models reuse one Transformer stack across several recurrent passes, refining hidden states before predicting a token. The LoopRPT paper proposes a reinforcement pre-training objective that scores those intermediate passes. On two Ouro checkpoints, the authors report higher accuracy at a given average loop count and earlier exits on tokens that require less computation.
Token rewards blur latent credit
LoopRPT builds on Ouro, a family of decoder-only language models that repeatedly applies the same stack of Transformer layers. Weight sharing keeps the parameter count fixed across passes, while each additional pass increases computation and latency.
| Property | Ouro-1.4B | Ouro-2.6B |
|---|---|---|
| Reused stack | 24 layers | 48 layers |
| Hidden size | 2,048 | 2,048 |
| Maximum recurrence | Four passes | Four passes |
| Pre-training data | Approximately 7.7 trillion tokens | Approximately 7.7 trillion tokens |
| Core components | Multi-head self-attention, RoPE, SwiGLU, and sandwich RMSNorm | |
Ouro predicts a halting probability during each recurrent pass. That probability determines whether computation stops or continues for a given token, allowing easy predictions to use less depth and difficult predictions to use more.
Conventional next-token training already propagates gradients through recurrent hidden states, but its supervision evaluates the resulting token distribution. Output-level reinforcement learning has the same credit-assignment problem: a final reward can influence the full trajectory without evaluating the contribution of each pass. LoopRPT adds a training signal tied to intermediate latent updates.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
