Recurrent Looped Transformer Carries Depth Across Tokens

Yifan Zhang has released a technical report and the RLT repository for the Recurrent Looped Transformer, an architecture that carries decoder state from one token to the next. A conventional causal Transformer sends each token through a fixed stack of layers. RLT extends the causal computation path as the sequence grows while keeping the decoder’s layer count fixed.

Zhang’s announcement on X described the release as a step toward superintelligence. The report lists reasoning gains, hardware speedups, and reinforcement-learning scaling as research goals. Its current evidence consists of an architecture specification and a small synthetic experiment with mixed results.

How depth accumulates

With decoder depth L_D, token position t sits at the end of a recurrent path containing t × L_D decoder-block applications. The model executes L_D decoder blocks for each new token, while the longest causal path grows linearly with sequence length. Attention and cache costs can still vary with context length.

The report’s phrase “infinite depth” refers to a temporal path that can keep extending as more tokens arrive. Every actual sequence has finite depth and compute. The report provides no measured evidence that a longer path improves reasoning.

Inside the recurrent loop

The reference configuration contains 48 encoder layers and 48 decoder layers. Compatible attention and feed-forward network weights are shared between the stages. Each decoder block also performs cross-attention over encoder memory, raising its floating-point operation count above that of an encoder block despite the equal layer counts.