vLLM's Hybrid HiSparse Triples Concurrent Requests on Million-Token Contexts

Serving models with million-token contexts on a single node has always run into the same wall: the KV cache balloons, GPU memory fills up, and the scheduler either preempts requests (paying full time-to-first-token again later) or blocks new ones. The vLLM team, working with Red Hat and Prime Intellect, just landed a memory-tier trick called Hybrid HiSparse that sidesteps this by letting individual requests keep decoding even after their KV stops fitting in HBM.

A pressure valve for KV cache

The core idea builds on sparse Multi-head Latent Attention (MLA), where an indexer picks the top-K most relevant tokens and attention only reads those. If the model only ever touches a small subset, the rest of the KV does not need to live on the GPU. The original HiSparse work exploited this by pushing all non-selected KV out to CPU memory, giving each request a fixed upper bound on GPU footprint.

Hybrid HiSparse adds a pressure-driven twist. KV starts on the GPU and stays there while there is room, then gives up residency page by page as the pool runs short. A request under memory pressure releases its coldest pages to host memory, keeps a small hot buffer of the rows the indexer keeps asking for, and continues decoding instead of being preempted or stalled.

Three residency states, one block pool

Each request moves between three states depending on how tight the pool is:

• Full residency: all sparse-MLA KV remains GPU-resident while completed prefix pages are proactively materialized in host memory.
• Mixed residency: the tail of the request stays on the GPU, older pages live only in CPU memory, and the rows the indexer wants from those pages sit in hot buffers. The block table holds real blocks and null placeholders side by side, and the tail is never evicted. One fused kernel resolves the top-K: resident tokens are read in place, hot tokens are read and their LRU entry refreshed, and a miss copies a single row from pinned host memory into an LRU slot.
• No residency: a new request reusing a prefix that only exists in CPU memory starts with placeholders and a hot page. Rows arrive as the indexer selects them, so you pay for what the model attends to rather than the whole history.

Hot buffers are not a separate allocation. A hot buffer page is an ordinary KV-cache block, leased from the same pool as the resident pages through vLLM's Hybrid Memory Allocator, taken when a request first needs one and returned when it does not. For rows in either a resident page or the hot buffer, the resolver hands HMA row IDs and HMA gathers them with one stride. A block freed by one request can immediately become hot-buffer capacity for another. The whole decode path stays CUDA-graph-capturable because nothing waits on a CPU-side decision.

The numbers on GLM 5.3

The team benchmarked GLM 5.3 on one 8x H200 node with a 13-turn OpenHands agent workload (74K-token first turn, ~750-token follow-ups). Same host budget, same admission limits, concurrency configured at 32:

Configuration	Concurrent running requests
KV offloading baseline (512 GiB host pool)	5-6
Hybrid HiSparse (384 GiB HiSparse + 128 GiB offload)	19-25

Roughly a 3-4x jump in sustained concurrency at long contexts, and the full 1M context length becomes reachable on this hardware for the first time. Both deployments used TP8, MTP3 speculative decoding, FP8 KV cache, and a 142K admission limit.

What it composes with

Because HiSparse is implemented as a residency policy over the shared HMA pool and a KV connector, the rest of vLLM's plumbing keeps working. Other cache groups still use normal prefix caching, transfer, and offloading. The indexer KV in particular is untouched by HiSparse: the standard OffloadingConnector can offload it independently with ordinary block-granular storage. P/D disaggregation imports and speculative decoding (MTP) also compose cleanly.

A few other details worth knowing:

• Hot buffers default to 2x top-K rows per request, which keeps hit rates high while holding the buffer size small.
• Since MLA KV is identical across TP ranks, the pinned host pool is allocated per DP replica and shared across its local TP ranks.
• HiSparse prepares for pressure before it arrives. When a cacheable prefix page is complete, HiSparse queues a copy to CPU memory while continuing to serve it from the GPU.
• Currently NVIDIA-only.

Where the tradeoffs bite

Hybrid HiSparse is no free lunch at every context length. Hot buffers add a fixed per-request GPU cost, so at short contexts ordinary residency can actually fit more requests. The advantage kicks in at longer contexts, where bounding sparse-MLA residency is what unlocks more concurrent requests. Larger hot buffers raise hit rates but eat into that concurrency headroom.

Speculative decoding has a wrinkle too: MTP can further limit concurrency because its hot buffers must accommodate all verification tokens at once. The team says this constraint is being worked on.

How to try it

Hybrid HiSparse is planned for general availability in vLLM v0.30. Until then, you can build from the pinned commit e8ef1e07bd on the neuralmagic fork and enable it via the new attention config flag:

vllm serve zai-org/GLM-5.3 \
  --tensor-parallel-size 8 \
  --kv-cache-dtype fp8 \
  --max-model-len 142000 \
  --enable-prefix-caching \
  --attention-config '{"hisparse_config":{"host_pool_gib":384}}' \
  --kv-transfer-config '{"kv_connector":"OffloadingConnector","kv_role":"kv_both","kv_connector_extra_config":{"spec_name":"TieringOffloadingSpec","cpu_bytes_to_use":137438953472}}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

The blog also ships an interactive concurrency calculator to estimate the benefit for your specific GPU, parallelism, and workload before committing to the rebuild. For workloads that look like long-context agent loops, with many concurrent conversations that keep growing, this is the kind of change that turns a memory-bound deployment into a compute-bound one.