SGLang's HiCache Slashes AI Response Times by 60% Across Distributed Clusters

SGLang has expanded HiCache so shared-prefix reuse remains consistent across context parallelism, pipeline parallelism, speculative decoding, prefill-decode disaggregation, and clusters that use multiple serving topologies.

Long-running agents repeatedly send earlier conversation turns with each request. During prefill, the model converts those tokens into attention keys and values, collectively called the KV cache. Reusing cached KV pages for a shared prefix reduces repeated computation and shortens time to first token, or TTFT.

The original HiCache design extended prefix caching from GPU memory into host memory and external storage. The latest technical report describes how SGLang now coordinates those cache tiers across distributed workers, including asynchronous transfers that finish at different times.

Astron pushes cache hits to 95%

iFLYTEK’s Astron MaaS team tested HiCache on GLM-5.1 and GLM-5.2 CodingPlan workloads using pipeline parallelism, separate prefill and decode workers, and multi-token prediction, which predicts several future tokens per decoding step. Input prompts typically range from 30,000 to 200,000 tokens, with some reaching 1 million.

Rank agreement is the hard part

A serving rank is one worker process participating in a distributed model deployment, usually on a dedicated GPU. The KV data for one logical prefix may span several ranks, model layers, memory tiers, and asynchronous transfer queues.

One logical request requires every participating rank to agree on the cached prefix length and the completed transfers it can consume. Divergent decisions can produce tensor-shape mismatches, collective communication hangs, or incomplete state for speculative decoding.

One prefix, many page owners

Context parallelism, or CP, divides the processing of a long input across ranks. Replicating every KV page on every CP rank consumes GPU and host capacity, so HiCache assigns each page to one owner.

Every CP rank maintains the same logical radix tree, an index that maps token prefixes to logical page IDs. A hit on pages A, B, and C therefore resolves to the same logical pages on every rank, while the underlying KV data remains distributed. Page q belongs to rank q mod N, where N is the number of CP ranks.

Before each attention layer, ranks gather their owned pages into a scratch buffer. Two alternating buffers allow the next layer’s transfer to overlap with the current layer’s computation. Supported configurations use GPU copy engines for the gather, reducing contention with model kernels that use streaming multiprocessors.

HiCache preserves page ownership when moving KV data to host memory and restoring it. Each recovered page returns to the rank responsible for contributing it to the next gather.

In a production comparison, GLM-5.2-NVFP4 ran on GB300 GPUs with four-way tensor parallelism, four-way context parallelism, and EAGLE multi-token speculative decoding.

Pipeline stages consume one event stream

Pipeline parallelism, or PP, assigns different model layers to different stages. Each stage stores KV data for its own layers and reaches cache-management checkpoints at a different wall-clock time.

Asynchronous transfers can complete in a different pattern on each stage. For example, the first stage may observe backup A as complete while backup B remains in flight. A later stage may find both complete when it checks its local queue. Applying those local observations would cause the stages to release different locks and diverge.

HiCache uses pp_sync to keep queue consumption consistent. The first pipeline stage selects the number of completed transfers to apply, combines that count across its tensor-parallel group, and sends the result through the pipeline. Every later stage consumes the same queue prefix without polling its local completion state.

External-storage prefetch follows the same ordering rule. Background threads calculate the minimum cache-hit length across ranks before publishing an acknowledgment, and schedulers consume those acknowledgments in sequence.

Draft models recover their full state

Speculative decoding uses a smaller draft model to propose tokens that the target model verifies. A target-model prefix hit can skip target prefill, but the draft model still needs KV data and any auxiliary state associated with the same prefix.

HiCache now stores the draft model’s KV cache, indexer buffers, and recurrent state alongside the target cache. Deployments can choose between two layouts:

• Packed: compatible draft layers are appended to the target model’s host-memory pools.
• Sidecar: draft state uses separate pools when tensor shapes or data types differ.

One external store serves multiple topologies

Production clusters often route short-context and long-context requests to separate buckets with different tensor-parallel and pipeline-parallel configurations. A topology-specific KV layout limits reuse because another configuration cannot directly reconstruct its local tensors from those objects.

HiCache’s unified layout partitions pages on a shared layer-and-head grid. Any compatible reader can load the required layer and head chunks, then assemble the local KV layout expected by its serving topology.

Object names combine the prefix hash with a namespace digest covering the model, data type, page size, and partition grid:

{page_hash}_{namespace_digest}_L{start_layer}-{end_layer}[_H{head_group}]

The current implementation supports the file backend with one multi-head attention or multi-head latent attention pool. Its scope excludes decode context parallelism and layer-sharded context parallelism.

• --hicache-storage-key-scheme unified selects unified object keys.
• --hicache-storage-head-partition controls head partitioning.
• --hicache-storage-layer-partition controls layer partitioning.

Decode workers reuse earlier turns

Prefill-decode disaggregation assigns prompt processing and token generation to separate workers. HiCache now gives each decode worker its own radix tree, allowing it to reuse prefixes retained from earlier conversation turns.

The decode worker reports a decode_prefix_len boundary to the prefill worker. Prefill then transfers only the suffix after that boundary, reducing redundant KV movement between the two worker pools.

A GLM-5.2-FP8 benchmark ran 32 concurrent conversations for 86 rounds, with contexts growing to 160,000 tokens. Enabling GPU and host prefix reuse on decode produced the following results:

Consensus failures become traceable

Distributed cache bugs often surface as stalled collectives far from the decision that caused ranks to diverge. SGLang’s rank_consensus_checker records selected scheduler calls and return values in logical order across ranks, then identifies the first inconsistent result and the responsible rank.

SGLANG_ENABLE_RANK_CONSENSUS_CHECKER=1

For agent and coding workloads, these changes preserve prefix-cache reuse as deployments add sharded context processing, pipelined layers, draft models, separate prefill and decode workers, and mixed-topology buckets backed by a shared external KV store. Implementation details and current code are available in the SGLang repository.