Tsinghua's NHA Beats Transformers on Long-Context Recall Without Quadratic Costs
A new hybrid attention design unifies linear RNN memory and sliding window attention under one softmax, matching Transformers with fewer full-attention layers.
Takeaways
- NHA unifies linear RNN memory and sliding-window attention under a single softmax, removing separate fusion parameters.
- Long-term context lives in fixed-size KV slots updated by a gated linear RNN, matching the format of local tokens.
- Inter-layer hybridization is controlled by one hyperparameter, the sliding window size, with no layer heterogeneity.
- NHA beats Transformer and hybrid baselines on recall-intensive, commonsense, and RULER long-context benchmarks at 340M and 1.3B scales.
- Pretrained Llama-3-8B, Qwen2.5-7B, and Qwen3-30B-A3B can be hybridized with only 10B or fewer finetuning tokens.
- Code and Triton kernel released at github.com/JusenD/NHA.
The bill for a Transformer's power keeps coming due at every token. Self-attention scales quadratically with sequence length, and once contexts stretch into tens of thousands of tokens the memory and latency curves start to bend in unpleasant ways. Linear attention and state space models fix the scaling by compressing history into a fixed-size state, but that compression consistently hurts recall on tasks that need to pull specific facts out of long inputs.
A team from Tsinghua, Shanghai AI Lab, HKUST (Guangzhou), and CUHK has proposed Native Hybrid Attention (NHA), a design that stops treating linear attention and softmax attention as two separate modules to be glued together. Both memory types share the same key-value format, and a single softmax decides, per token and per head, how much to lean on each. The code is on GitHub.
Why bolting linear and full attention together has been awkward
Hybrid architectures have been the dominant response to the recall-versus-efficiency tradeoff, and they generally come in two flavors:
- Intra-layer hybrids (Infini-attention, LoLCATs, Liger, Griffin) compute a linear attention output and a local sliding-window attention output separately, then merge them with a fixed or learned scalar coefficient.
- Inter-layer hybrids (Jamba, Zamba, Samba, GoldFinch) stack different layer types, for example one full-attention layer for every seven linear layers.
Both approaches carry structural baggage. Intra-layer fusion runs two attention operations per query and picks the global-versus-local balance outside the attention computation, so the tradeoff is uniform across tokens for a given query and can't respond to the actual similarity between the query and the keys. Inter-layer hybrids force a rigid choice per layer, require managing heterogeneous blocks, and finding a good configuration generally means retraining each candidate.