LMSYS Rebuilds SGLang's Cache to Finally Support Hybrid AI Models
SGLang's Unified Radix Cache replaces a growing matrix of specialized cache classes with one composable tree, unlocking proper prefix caching for hybrid models like DeepSeek-V4 and Inkling.
- One tree for all hybrid models: SGLang's Unified Radix Cache replaces separate RadixCache, MambaRadixCache, and SWARadixCache classes with a single composable tree.
- Composable components: FULL, SWA, and MAMBA components enforce their own reuse rules via a voting mechanism, finding the deepest prefix boundary all components accept.
- Massive HiCache throughput gains: With L3 (Mooncake), DeepSeek-V4-Flash hits 145.5K effective input tokens/s vs 9.4K with GPU-only cache at 98% hit rate.
- Session-aware eviction: Attaching a session_id improves TTFT by 2.9–16.6% on SWE-bench agent workloads vs standard LRU eviction.
- Experimental Rust tree core: An opt-in Rust prototype reduces TTFT by up to 42% on long sliding-window attention workloads.
- Available now: Enable with
SGLANG_ENABLE_UNIFIED_RADIX_TREE=1; session eviction via--enable-session-radix-cache.
Prefix caching is one of the highest-leverage optimizations in LLM serving: when multiple requests share the same token prefix, you skip recomputing their key-value (KV) states and serve them from cache instead. For pure transformer models, this is straightforward. But the wave of hybrid architectures -- models that mix full attention, sliding window attention (SWA), and recurrent Mamba/SSM layers -- has quietly broken the assumption that one caching rule fits all.
Unified Radix Cache, a new design from the LMSYS team, is SGLang's answer to that problem. Instead of maintaining a growing zoo of specialized cache classes, it consolidates everything into a single token-keyed radix tree with composable, per-architecture reuse rules attached as pluggable components.
Why hybrid models break the old approach
A radix tree (also called a prefix tree) is the data structure SGLang uses to track which token sequences have cached KV states. When a new request arrives, the tree finds the longest matching prefix and hands the scheduler the memory locations to reuse. Under full attention, this is clean: once a prefix is cached, it stays valid forever as the conversation grows.
Hybrid models shatter that clean rule. Consider a request processed by a model like DeepSeek-V4 or Inkling:
- Full attention KV is reusable across the entire matched prefix.
- Sliding window attention (SWA) KV only covers a trailing window of tokens -- older slots are stale.
- Mamba/recurrent states are valid only at an exact checkpoint position and cannot be partially reused.
These values share the same token prefix, but not the same reusable boundary.
Forcing a single boundary either throws away valid cache hits or, worse, permits invalid reuse that produces incorrect outputs. The previous SGLang approach handled this by building separate cache classes -- RadixCache, MambaRadixCache, SWARadixCache -- each duplicating the matching, insertion, locking, and eviction logic.
These implementations shared a large amount of logic but were maintained as separate, diverged copies, leading to code duplication, inconsistent behavior, and a high maintenance burden when extending cache functionality to new model types.
This problem is not limited to SGLang: prefix caching only works correctly for pure full-attention models in many frameworks. Any model using sliding window attention, Mamba/SSM layers, or mixed attention types silently falls back to full prompt recomputation on every request, making multi-turn conversations unusably slow for the majority of modern open-weight models.