---
title: "Tsinghua's NHA Beats Transformers on Long-Context Recall Without Quadratic Costs"
subtitle: "A new hybrid attention design unifies linear RNN memory and sliding window attention under one softmax, matching Transformers with fewer full-attention layers."
url: "https://alphasignal.ai/news/tsinghua-s-nha-beats-transformers-on-long-context-recall-without-quadratic-costs"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-04-15T00:00:00.000Z"
updated: "2026-08-23T21:23:56.591Z"
type: "Paper"
topics:
  - "Llms"
  - "Training Infra"
subtopics:
  - "Long Context"
  - "Scaling Laws"
upvotes: 2394
source_url: "https://arxiv.org/abs/2510.07019"
comments: 0
truncated: true
---

# Tsinghua's NHA Beats Transformers on Long-Context Recall Without Quadratic Costs

> A new hybrid attention design unifies linear RNN memory and sliding window attention under one softmax, matching Transformers with fewer full-attention layers.

The bill for a Transformer's power keeps coming due at every token. Self-attention scales quadratically with sequence length, and once contexts stretch into tens of thousands of tokens the memory and latency curves start to bend in unpleasant ways. Linear attention and state space models fix the scaling by compressing history into a fixed-size state, but that compression consistently hurts recall on tasks that need to pull specific facts out of long inputs.

A team from Tsinghua, Shanghai AI Lab, HKUST (Guangzhou), and CUHK has proposed [Native Hybrid Attention (NHA)](https://arxiv.org/abs/2510.07019), a design that stops treating linear attention and softmax attention as two separate modules to be glued together. Both memory types share the same key-value format, and a single softmax decides, per token and per head, how much to lean on each. The [code is on GitHub](https://github.com/JusenD/NHA).

## Why bolting linear and full attention together has been awkward

Hybrid architectures have been the dominant response to the recall-versus-efficiency tradeoff, and they generally come in two flavors:

* **Intra-layer hybrids** (Infini-attention, LoLCATs, Liger, Griffin) compute a linear attention output and a local sliding-window attention output separately, then merge them with a fixed or learned scalar coefficient.
* **Inter-layer hybrids** (Jamba, Zamba, Samba, GoldFinch) stack different layer types, for example one full-attention layer for every seven linear layers.

Both approaches carry structural baggage. Intra-layer fusion runs two attention operations per query and picks the global-versus-local balance outside the attention computation, so the tradeoff is uniform across tokens for a given query and can't respond to the actual similarity between the query and the keys. Inter-layer hybrids force a rigid choice per layer, require managing heterogeneous blocks, and finding a good configuration generally means retraining each candidate.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Native Hybrid Attention for Efficient Sequence Modeling](https://arxiv.org/abs/2510.07019)
- [NHA GitHub Repository](https://github.com/JusenD/NHA)
- [NHA HTML paper (arxiv.org)](https://arxiv.org/html/2510.07019)