LMSYS Rebuilds SGLang's Cache to Finally Support Hybrid AI Models

Prefix caching is one of the highest-leverage optimizations in LLM serving: when multiple requests share the same token prefix, you skip recomputing their key-value (KV) states and serve them from cache instead. For pure transformer models, this is straightforward. But the wave of hybrid architectures -- models that mix full attention, sliding window attention (SWA), and recurrent Mamba/SSM layers -- has quietly broken the assumption that one caching rule fits all.

Unified Radix Cache, a new design from the LMSYS team, is SGLang's answer to that problem. Instead of maintaining a growing zoo of specialized cache classes, it consolidates everything into a single token-keyed radix tree with composable, per-architecture reuse rules attached as pluggable components.

Why hybrid models break the old approach

A radix tree (also called a prefix tree) is the data structure SGLang uses to track which token sequences have cached KV states. When a new request arrives, the tree finds the longest matching prefix and hands the scheduler the memory locations to reuse. Under full attention, this is clean: once a prefix is cached, it stays valid forever as the conversation grows.

Hybrid models shatter that clean rule. Consider a request processed by a model like DeepSeek-V4 or Inkling:

• Full attention KV is reusable across the entire matched prefix.
• Sliding window attention (SWA) KV only covers a trailing window of tokens -- older slots are stale.
• Mamba/recurrent states are valid only at an exact checkpoint position and cannot be partially reused.

These values share the same token prefix, but not the same reusable boundary. Forcing a single boundary either throws away valid cache hits or, worse, permits invalid reuse that produces incorrect outputs. The previous SGLang approach handled this by building separate cache classes -- RadixCache, MambaRadixCache, SWARadixCache -- each duplicating the matching, insertion, locking, and eviction logic. These implementations shared a large amount of logic but were maintained as separate, diverged copies, leading to code duplication, inconsistent behavior, and a high maintenance burden when extending cache functionality to new model types.

This problem is not limited to SGLang: prefix caching only works correctly for pure full-attention models in many frameworks. Any model using sliding window attention, Mamba/SSM layers, or mixed attention types silently falls back to full prompt recomputation on every request, making multi-turn conversations unusably slow for the majority of modern open-weight models.

One tree, composable rules

Unified Radix Cache separates two concerns that were previously tangled together: prefix identity (which token sequence are we talking about?) and reuse validity (is this cached value actually safe to reuse for this architecture?). A single token-keyed radix topology answers the first question. Pluggable TreeComponent objects answer the second.

Three components ship today:

• FULL -- always present, provides path reuse for standard attention KV across the entire matched prefix.
• SWA -- added for sliding window attention, requires a contiguous trailing window; older slots become tombstones in the shared topology.
• MAMBA -- added for recurrent layers, requires one exact checkpoint at the reusable frontier and copies shared state into a private request slot before mutation.

For example, DeepSeek-V4 composes FULL and SWA, Kimi-K3 composes FULL and MAMBA for its KDA recurrent state, and Inkling composes all three components on the same tree. A new model family can reuse an existing component composition.

During prefix matching, the tree walks the canonical FULL path and treats each visited node as a candidate boundary. Every active component creates a validator, and the reusable boundary only advances when all validators agree. A component can reject a node without stopping traversal -- the walk continues deeper, but the safe boundary stays at the last node that passed every check. This is called component voting: the tree reaches depth N, but the scheduler only gets the prefix up to the deepest node that every component signed off on.

HiCache goes hybrid-native

SGLang's HiCache is a three-tier KV cache hierarchy inspired by CPU cache design. It organizes GPU memory as L1, host memory as L2, and distributed storage as L3. Previously, HiCache only worked with standard full-attention models. Unified Radix Cache makes it native to the component lifecycle: when a cached value moves from GPU to host RAM to an external distributed store like Mooncake, it carries the same prefix identity and reuse rules with it.

The design also introduces a distinction between components and sidecars. Components define reuse semantics. Sidecars are auxiliary pools (like compressed KV variants) that simply follow a component's index space without voting on the reusable boundary or adding complexity to the tree. For DeepSeek-V4, the C4 and C128 compressed KV pools register as sidecars, keeping the tree clean while still moving across memory tiers.

The benchmark results for multi-turn workloads are striking. On DeepSeek-V4-Flash, the L3 configuration keeps the hit rate near 98%, holds average TTFT below 9 seconds, and reaches 145.5K effective input tokens/s, compared with 9.4K for L1 only and 14.3K for L1 plus L2. On Inkling-Small, L3 finishes at a 96.8% hit rate and 1.23-second TTFT while reaching 67.1K effective input tokens/s, compared with 15.5K for L1 and 21.1K for L1 plus L2. The gains come from preserving reusable prefixes after GPU capacity is exhausted -- without L3, cache hit rates collapse as conversation history grows beyond what fits on-device.

Session-aware eviction for agentic workloads

Standard LRU eviction has a blind spot: it knows what was accessed recently, but not which prefixes belong to active ongoing sessions that are about to send another message. Under memory pressure, it can evict an active session's GPU KV while retaining unrelated cached entries from sessions that are already done.

Unified Radix Cache adds session-aware eviction directly into the shared tree. Applications attach a stable session_id to each request. After a request completes, the cache registers the reusable region for that session across all components. These references change eviction order rather than pinning memory: FULL orders candidates by whether they are referenced, their session reference count, and the configured base eviction priority.

The SWE-bench agent trajectory results show the impact clearly. Relative to the ordinary HiRadixCache baseline, the session-aware Unified Radix Cache configuration observes 11.0% and 2.9% lower TTFT for DeepSeek-V4-Pro at batch sizes 128 and 256. Qwen3.5-397B-A17B observes 13.5% and 16.6% lower TTFT at batch sizes 32 and 64.

An experimental Rust core

As shared prefixes grow long, tree bookkeeping -- traversal, lock accounting, LRU updates, eviction scans -- adds latency on the scheduler's critical path. The team built an experimental opt-in Rust implementation of the tree core to address this. Rust owns the radix topology, per-component lock accounting, intrusive LRU lists, and eviction walks. Python remains the owner of request-to-token mappings and physical KV allocation.

The Rust prototype records the largest reduction on the SWA workload: TTFT is 38% lower across all 200 turns and 42% lower over turns 176 to 200. Full attention records 10% lower TTFT overall and 18% lower over the final 25 turns. The hybrid SSM workload sees more modest gains because the GPU forward pass dominates total latency there.

This is currently an L1-only prototype and does not yet support HiCache. A follow-up RFC (#32710) defines the target ownership boundary for a production-ready version.

How to enable it

Unified Radix Cache is available in SGLang today. Enable it with a single environment variable:

export SGLANG_ENABLE_UNIFIED_RADIX_TREE=1

Session-aware eviction is enabled with the --enable-session-radix-cache server flag, and applications attach a session_id to requests. HiCache flags (--enable-hierarchical-cache, --hicache-ratio, --hicache-storage-backend, etc.) work as before and are now natively composable with hybrid model components. The Rust tree core is opt-in and separate.

The practical use-cases that benefit most are:

• Multi-turn agentic workloads (coding agents, SWE-bench-style trajectories) where session prefixes grow across many rounds
• High-concurrency serving of hybrid models like DeepSeek-V4, Kimi-K3, or Inkling where the old per-model cache classes created correctness and maintenance headaches
• Deployments with HiCache that want to extend hierarchical caching to Mamba or SWA models without building custom cache logic
• Long-context RAG pipelines that reuse the same document context across many requests

The deeper significance here is architectural. In practice, frameworks like vLLM ended up with separate managers: a normal KV cache, a Vision Encoding Cache, a Mamba Cache, etc. This works but is fragile and hard to extend. Unified Radix Cache is a direct answer to that fragility: a composable foundation that can absorb new hybrid architectures without forking the tree. As models continue mixing attention types in novel combinations, having a single extensible cache abstraction becomes less of a nice-to-have and more of a prerequisite for keeping serving infrastructure maintainable.