---
title: "SANTA Slashes AI Memory Costs by 90% at Long Contexts Without Losing Accuracy"
subtitle: "A new paper introduces SANTA, a stochastic sparse attention method that cuts value-cache memory reads at decode time, delivering up to 1.5× kernel speedup with no accuracy loss at 32k-token contexts."
url: "https://alphasignal.ai/news/santa-slashes-ai-memory-costs-by-90-at-long-contexts-without-losing-accuracy"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-05-01T00:00:00.000Z"
updated: "2026-06-11T02:33:28.276Z"
type: "Paper"
topics:
  - "Infra"
  - "Gpus"
subtopics:
  - "Inference Optimization"
  - "Kernels"
  - "Quantization"
upvotes: 2886
source_url: "https://arxiv.org/abs/2605.01910"
comments: 0
truncated: true
---

# SANTA Slashes AI Memory Costs by 90% at Long Contexts Without Losing Accuracy

> A new paper introduces SANTA, a stochastic sparse attention method that cuts value-cache memory reads at decode time, delivering up to 1.5× kernel speedup with no accuracy loss at 32k-token contexts.

Every token an LLM generates at long context requires the GPU to read the entire key-value (KV) cache , a cost that grows linearly with sequence length. For Llama-3.1-8B at a 32k-token context, that means streaming roughly 128 MB of key and value vectors **per layer, per token**. At that scale, the bottleneck is no longer arithmetic; it's memory bandwidth. A new research paper proposes a fundamentally different approach to this problem: instead of reading all the values, sample a small random subset and average them.

The paper, titled ["Stochastic Sparse Attention for Memory-Bound Inference"](https://arxiv.org/abs/2605.01910), introduces **SANTA (Stochastic Additive No-mulT Attention)**, a drop-in replacement for the value aggregation step in standard attention. The authors , Kyle Lee, Corentin Delacour, Kevin Callahan-Coray, Kyle Jiang, Can Yaras, Samet Oymak, Tathagata Srimani, and Kerem Y. Camsari , also release their [CUDA kernels as open source](https://github.com/OPUSLab/SANTA).

## Why the value stage is the right place to attack

Standard scaled dot-product attention (SDPA) has two expensive memory stages during decoding: computing attention scores via `qK^T` (the score stage), and then aggregating value vectors weighted by those scores (the value stage). Both require reading the full KV cache. Prior work like FlashAttention and FlashDecoding optimized _how_ you read everything; SANTA asks whether you need to read everything at all.

Autoregressive decoding becomes bandwidth-limited at long contexts, as generating each token requires reading all key and value vectors from the KV cache. The key insight in SANTA is that the post-softmax attention weights form a probability distribution over keys. That distribution can be _sampled_. If you draw S indices from it and average the corresponding value rows, you get an unbiased estimate of the full weighted sum , with no multiplications required after the softmax.

## The math: sampling as Monte Carlo attention

In standard attention, the output is `AV` , the attention weight matrix multiplied by all value rows. SANTA replaces this with:

```
# SANTA: sample S indices from the softmax distribution
# and average the corresponding value rows
import torch
def santa_attention(q, K, V, S):
    # Compute attention scores and softmax
    scores = (q @ K.T) / (K.shape[-1] ** 0.5)
    A = torch.softmax(scores, dim=-1)  # shape: [n_k]
    # Sample S indices from the categorical distribution
    indices = torch.multinomial(A, num_samples=S, replacement=True)
    # Gather sampled value rows and average
    sampled_V = V[indices]  # shape: [S, d_k]
    return sampled_V.mean(dim=0)  # unbiased estimate of AV
```

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Stochastic Sparse Attention for Memory-Bound Inference (arXiv)](https://arxiv.org/abs/2605.01910)
- [SANTA HTML Paper (arXiv)](https://arxiv.org/html/2605.01910)
- [OPUSLab/SANTA GitHub Repository](https://github.com/OPUSLab/SANTA)