South China University's LCA Slashes DeepSeek-V2-Lite Memory 90% at 128K Context
A new attention mechanism condenses context directly inside DeepSeek-style latent space, cutting KV cache by 90% and speeding prefill 2.5x at 128K tokens.
PRO- Latent-Condensed Attention condenses tokens inside MLA's low-dim latent space, jointly cutting KV cache and compute
- Semantic latents use query-aware weighted pooling; positional RoPE keys use hard anchor selection to preserve geometry
- 2.5x prefill speedup and 90% KV cache reduction at 128K on DeepSeek-V2-Lite, matching MLA accuracy
- Length-independent error bound proven; empirical key/value deviations stay under 5% across 4K-128K contexts
- Zero new parameters, integrates with just 1,000 fine-tuning steps; ships with a custom Triton kernel
- Weak spot is multi-document QA where attention concentrates on a few tokens that get averaged inside their group
Long-context inference has two well-known bottlenecks that people usually treat as separate problems. The KV cache grows linearly with sequence length, eating GPU memory, and self-attention scales quadratically with compute. A new paper from South China University of Technology and Pengcheng Laboratory proposes Latent-Condensed Attention (LCA), which attacks both at once by compressing tokens directly inside the low-dimensional latent space that Multi-head Latent Attention (MLA) already uses.
The reported numbers are worth pausing on: up to 2.5x prefill speedup and 90% KV cache reduction at 128K context on DeepSeek-V2-Lite, with average scores on LongBench-E and RULER that essentially match the original MLA model. On RULER at 128K, LCA actually edges out the baseline (24.38 vs 23.96).
Why MLA alone fell short
MLA, the attention variant popularized by DeepSeek-V2/V3, projects each token into a shared low-dimensional latent vector and caches only that plus a small positional key. This dramatically shrinks per-token cache. But it still keeps every one of the L tokens around and runs dense attention over all of them, so the quadratic compute cost stays put.
The natural next move is to bolt on a sparse attention method like MInference or FlexPrefill. The catch is subtle but important: most existing sparse methods operate on attention scores between original queries and keys, which require full-dimensional representations, so when applied to MLA they must first reconstruct full KV matrices before sparsifying. That reconstruction throws away the memory benefit MLA was designed to give you. In the authors' benchmarks, adapting MInference to MLA at 128K drops the RULER score from 23.96 all the way to 4.34.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
