Latent Sparse Attention Speeds Up 10M-Token AI Contexts by 9x

AI agents are getting better at working on tasks that span hours and involve entire codebases, document libraries, and long histories of interactions.

But the longer they work, the more expensive it becomes for the underlying models to process their growing context. Even sparse attention, designed to reduce this cost, has scaling problems of its own.

Researchers have developed Latent Sparse Attention (LSA), a technique that could make million-token contexts much more practical. Today, we examine the results, the tradeoffs, and what developers can already test.

How latent sparse attention tackles the search bottleneck in long-context AI

AI agents (particularly coding agents) accumulate millions of tokens of context as they work with large code repositories and document libraries. At that scale, handling the attention values of the underlying model becomes tedious and expensive.

To address this bottleneck, researchers at Prefrontal AI and ETH Zurich have developed Latent Sparse Attention (LSA), a new technique that cuts the compute costs of processing long contexts in LLMs.

LSA is among a growing range of techniques for optimizing the attention mechanism in LLMs. However, it addresses a bottleneck in other sparse attention methods, which limits how many previous tokens a model examines.

LSA organizes attention values into clusters and groups to make it easier to search through them when mapping new queries. For example, in experiments on 10-million-token tasks, it reduced time to first token by more than 9 times a full-attention model and 2.2 times against a model using a popular sparse attention mechanism.

For developers building long-running agents and LLM applications that process large contexts, LSA offers a way to reduce inference latency.