---
title: "Microsoft's CLSA Slashes Long-Context AI Decoding by 7.6x With One Trick"
subtitle: "Microsoft Research's CLSA architecture computes sparse attention routing once and shares it across all decoder layers, hitting 17.1x throughput gains at 128K context with near-zero quality loss."
url: "https://alphasignal.ai/news/microsoft-s-clsa-slashes-long-context-ai-decoding-by-7-6x-with-one-trick"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-06-04T17:54:04.000Z"
updated: "2026-06-15T00:55:08.855Z"
type: "Paper"
topics:
  - "Llms"
  - "Infra"
subtopics:
  - "Long Context"
  - "Mixture Of Experts"
  - "Inference Optimization"
upvotes: 48
source_url: "https://arxiv.org/abs/2606.06467"
comments: 0
truncated: true
---

# Microsoft's CLSA Slashes Long-Context AI Decoding by 7.6x With One Trick

> Microsoft Research's CLSA architecture computes sparse attention routing once and shares it across all decoder layers, hitting 17.1x throughput gains at 128K context with near-zero quality loss.

Long-context inference is quietly becoming one of the most expensive problems in production AI. As reasoning models like o1 and DeepSeek-R1 generate thousands of tokens of intermediate chain-of-thought before producing an answer, the cost of attending over a growing KV cache at every single layer compounds fast. A team from **Microsoft Research and Tsinghua University** has now published a paper proposing a surprisingly clean fix: compute the expensive routing step once, and reuse it everywhere.

The work, titled ["You Only Index Once: Cross-Layer Sparse Attention with Shared Routing" (CLSA)](https://arxiv.org/abs/2606.06467), tackles a bottleneck that has quietly undermined most prior sparse attention approaches. The result is a 4B-parameter architecture that achieves **7.6x decoding speedup** and **17.1x overall throughput improvement** at 128K context, while matching dense attention on nearly every quality benchmark.

## The problem no one fully solved

Sparse attention -- the idea of having each token only attend to a small, relevant subset of the KV cache instead of every prior token -- has been an active research area for years. But in practice, it has faced a stubborn tradeoff:

* **Block-sparse methods** (like NSA and MoBA) group tokens into chunks and attend to whole blocks at a time. They map well to GPU memory access patterns and deliver real wall-clock speedups, but the coarse granularity introduces quality loss -- nearby tokens in the same block can be semantically unrelated.
* **Token-sparse methods** (like DeepSeek-V3's DSA) select individual high-relevance tokens with fine-grained top-k routing. They preserve quality better, but the top-k operation itself is the problem. Top-k routing over the full cache remains expensive because it must be recomputed independently at every layer during decoding -- and top-k is poorly suited to GPU tensor cores, which thrive on dense matrix multiplications.

The result: token sparse methods are usually more accurate yet deliver limited end-to-end speedup because top-k routing over the full cache remains expensive. At 128K context, the paper's latency analysis shows that an unamortized top-k pass can take as long as substantial dense attention work, despite involving far fewer arithmetic operations.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [You Only Index Once: Cross-Layer Sparse Attention with Shared Routing](https://arxiv.org/abs/2606.06467)
- [You Only Cache Once: Decoder-Decoder Architectures for Language Models](https://arxiv.org/abs/2405.05254)
- [YOCO++: Enhancing YOCO with KV Residual Connections for Efficient LLM Inference](https://arxiv.org/abs/2604.13556)
