Looped transformers: three different machines

You've probably heard the buzz around Looped transformers, but which one? Over the past month, that term was applied to three fundamentally different machines.

Before diving into them, let's agree that a LLM writes one token at a time, and a transformer is the layer stack that turns the tokens you already have into the next one. A looped design reuses some of that stack instead of adding more unique layers.

But there are different reuse mechanisms, and if you implement the wrong one, you will either waste a week of training compute or trigger out-of-memory errors by budgeting a 3B-weight model as a standard 3B decode.

Next-token loss is measured in nats, the standard base(e) cross-entropy loss, where lower signifies better prediction. In an independent evaluation at 140M parameters across 500M tokens, the viral Recurrent Looped Transformer underperformed a standard baseline (3.6884 vs. 3.6420 nats) while burning roughly 20x the GPU-hours.

To evaluate any looped model, apply a three-part test: classify its mechanism: depth, time, or flow, verify whether extra inference steps actually buy higher quality, and determine whether its runtime profile belongs in production or gets rejected.

Then we all saw the Astra rumor that made the collision feel urgent. The Information reported that OpenAI's Astra uses "recurrent depth," with a cost and performance upside and a worry that it hides thinking.

Jakub Pachocki, who leads research at OpenAI, answered the same morning: "The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4." Chain-of-thought monitoring, he wrote, is "trending in a negative direction, for reasons not contingent on architecture changes."

An OpenAI safety researcher quote-tweeting him said a frontier-scale recurrent (or otherwise unmonitorable) language model would be one of the darkest in the current AI era, and "this day is not today."

For now, Astra remains unconfirmed speculation: bounded by Pachocki's factor-of-two graph depth constraint, and a safety researcher at OpenAI wrote that this day is not today. I'd keep any loop count out of an Astra runbook.

What follows is the three-way check, the extra-compute test, Nanbeige 4.2's two-pass KV bill, why the time-loop ran at 8.4K tokens/s against 124K on a 5090, and when a 5-7 million flow net is the right sidecar.

What is looped transformer?

A "looped transformer" is one of three machines: extra layer passes on one token, hidden state carried token to token, or a tiny net that denoises a grid from noise.

In early September, Sebastian Raschka drew the initial distinction: reusing layers across depth versus maintaining recurrent state across time. If a README simply advertises a "looped" architecture and you can't categorize which mechanism it uses, you don't actually know what you are implementing.

A concrete check: