Microsoft Finds a 2023 Trick Beats Two Years of Linear Attention Research

A quiet finding from Microsoft's Applied Sciences Group is likely to annoy a lot of people who spent the last two years post-training linear attention variants. The claim, backed by benchmarks across models from 1.3B to 70B parameters, is that a stock trick from 2023 called Sliding Window Attention with attention sinks matches or beats those methods on short-context reasoning and crushes them on long-context tasks, with zero fine-tuning required.

The paper, Sliding-window beats linear attention, argues that the entire linearization research direction has been comparing itself to the wrong baseline. Swap in the correct baseline, and the case for post-training a pretrained LLM into a linear attention model largely falls apart.

The quadratic tax nobody wants to pay

Standard self-attention has a well-known scaling problem. Every new token's keys and values must be added to the KV cache, and they contribute to an ever-growing memory and compute cost. That is why inference on long contexts gets expensive fast and why hosting frontier models is a constant battle against memory.

Linear attention methods reduce time and memory complexity from quadratic to linear. They replace the softmax similarity with a kernel that factorizes, so instead of storing every past key and value, you keep a fixed-size running state. Inference becomes constant memory and constant time per token, which sounds like a free lunch.

The catch is that training linear attention transformers from scratch is expensive and the kernels are poorly supported. So the field converged on retrofitting: take a pretrained Llama or Qwen, swap in a linear attention block, and post-train it to recover the lost quality. LoLCATs showed that this could be done with as little as 40M tokens by combining Hedgehog and Sliding Window Attention. A whole zoo of methods followed: SUPRA, Liger-GLA, MOHAWK, Mamba in the Llama, Llamba, QRWKV6/7, and others.