How Local Layers Give NoPE Attention a Sense of Distance

Hybrid language models can retain token-distance information even when some global attention layers omit explicit positional encoding. A new paper from Zyphra Research attributes that behavior to local mixing: sliding window attention and gated linear attention make nearby residual states more similar, and trained global attention heads convert that geometry into logits that favor recent tokens.

The finding helps explain why architectures that interleave local layers with global NoPE attention can model order effectively. It also identifies window size as a consequential hyperparameter, linking smaller local windows to stronger recency signals and lower validation loss in the authors’ experiments.

Where order can hide

• Sliding window attention (SWA) limits each token to a fixed number of recent tokens.
• Gated linear attention updates a recurrent state while controlling how quickly older information decays.
• Global NoPE attention attends across the available context without applying an explicit position-dependent transformation to its queries and keys.
• The residual stream carries each layer’s hidden states into subsequent attention and MLP blocks.

Without position-dependent inputs or masks, self-attention is permutation-equivariant: shuffling the input tokens produces the same shuffle in the output. Decoder models usually add order through a causal mask and an encoding such as RoPE, which rotates query and key vectors according to token position.

A causal mask already supplies limited ordering information because each token can access only earlier positions. Pure NoPE decoders can learn from that asymmetry, although the resulting signal has weak distance resolution as sequences grow. Hybrid models provide another source of order because every global layer receives hidden states previously transformed by local mixers.