MSAR brings multi-head attention to the residual stream

Residual connections in most PreNorm Transformers carry each block’s output forward through repeated addition. The MSAR paper replaces that accumulation with multi-head attention over earlier layer outputs. Across 18B and 28B Mixture-of-Experts experiments, the authors report a 1.58x iso-loss compute speedup over standard residuals.

That ratio corresponds to roughly 37% less training compute to reach the same modeled loss under the paper’s scaling-law fits. Wall-clock throughput, memory use, kernel efficiency, and end-to-end training cost require separate measurement.

Why addition blurs depth

A PreNorm Transformer block typically follows an update such as xl+1 = xl + Fl(LN(xl)). After many layers, xl contains an unweighted sum of earlier contributions. The number of terms grows linearly with depth, and each earlier output becomes a smaller part of the aggregate presented to later blocks.

Later layers can transform the accumulated state, but they cannot selectively retrieve an earlier layer output after standard addition has merged it into the stream. This limits depth-wise routing: every block receives the same aggregate history regardless of which earlier representations would be most useful.

Attention Residuals, or AttnRes, introduced a learned softmax distribution over preceding layer outputs. Each layer can assign more weight to useful depths, including the original token embedding. The authors describe this correspondence as time-depth duality. Sequence attention retrieves selected positions from token history; depth attention retrieves selected representations from layer history.

The failed multi-head attempt

AttnRes uses a single pseudo-query per layer, making its depth retrieval analogous to single-head attention. Earlier experiments divided that operation into multiple heads, but performance declined. Those results suggested that channel groups were learning similar depth mixtures and gaining little from separate heads.

MSAR gives each head a distinct contextual view of the token sequence before it attends over depth. The resulting diversity comes from sequence context, allowing heads to specialize without maintaining several full residual streams between layers.