LMSYS Rebuilds SGLang's Memory to Handle Hybrid Models 26% Faster
SGLang's new unified memory arena removes fixed KV partitions for hybrid models, dynamically sharing GPU memory across full-attention, sliding-window, and recurrent state.
- SGLang ships Unified Memory, a single GPU arena shared by full-attention KV, sliding-window KV, and recurrent state.
- Removes the
--swa-full-tokens-ratioand--mamba-full-memory-ratiostartup knobs that forced workload profiling before serving. - Per-token entry layout packs all layers' rows together, enabling speculative draft KV and DSA indexer keys in one allocation.
- On GPT-OSS-20B, runs ~1,000 concurrent requests vs 618 for separate pools, with 14% higher throughput and TTFT dropping from 43s to 6s.
- Three-step admission ladder (no move, compact ends, relayout) searches arena layouts to fit joint demand.
- Experimental; hybrid models only, enabled via
--enable-unified-memory. See usage guide.
LMSYS has added Unified Memory to SGLang for hybrid architectures including GPT-OSS, Qwen3.5, and Kimi K3. These models combine full attention, sliding-window attention, and recurrent or linear-attention layers, each with a different memory profile. SGLang previously required operators to split GPU high-bandwidth memory (HBM) among those state types before launch. The experimental allocator lets them share one arena and adjusts the layout as traffic changes, reducing idle capacity and avoidable evictions.
Why fixed pools waste HBM
Autoregressive attention stores key and value (KV) tensors from earlier tokens to compute the next token. Full-attention layers retain KV for the entire context, so their memory use grows with sequence length. Sliding-window layers keep a bounded suffix. Recurrent and linear-attention layers maintain a fixed-size state for each active request, making their memory use depend mainly on concurrency.
SGLang's previous allocator carved HBM into separate pools configured through --swa-full-tokens-ratio and --mamba-full-memory-ratio. Long-context traffic could exhaust the full-attention pool while window KV and recurrent-state capacity remained idle. Short, highly concurrent traffic could fill the bounded-state pools while full-attention capacity sat unused. Free memory could not cross those pool boundaries.
One arena, mixed layouts
Enabling --enable-unified-memory places full-attention KV at the high end of one arena and grows it downward. Sliding-window KV and recurrent state start at the low end and grow upward. Lazy compaction preserves a free region between them. The two ratio flags remain available as initial placement hints without imposing fixed capacity partitions after startup.
The allocator represents one token, or one recurrent-state slot, as an entry containing rows for every participating layer. A page consists of a contiguous run of entries, so allocation, eviction, and page movement update all layers together. Rows within an entry can retain different shapes and data types. That layout allows speculative draft-model KV to share storage and lifecycle rules with target-model KV, even when their data types differ. It also keeps Kimi K3.1's Dynamic Sparse Attention index keys alongside the Multi-head Latent Attention KV they reference.
Admission becomes a packing problem
An incoming request may require full-attention pages, sliding-window pages, and a recurrent-state slot. SGLang therefore evaluates the available layout through three increasingly expensive steps:
- Reuse in place: fill existing holes and extend either end band without moving data.
- Compact the ends: pack allocations within the high and low bands to remove internal gaps.
- Relayout: move eligible allocations into the space between the compacted end bands.
Virtual page IDs keep logical references stable while SGLang copies a page and remaps its physical location. A page-use ledger records model forward passes, HiCache copies, and network transfers that still reference each page. When a request releases a page, SGLang marks it logically free and withholds its physical address from new allocations until the ledger records no outstanding users.
Where the H200 gained headroom
LMSYS compared unified memory with SGLang's default separate-pool configuration on one NVIDIA H200. The tested workloads showed near-parity under moderate pressure and higher request capacity when a fixed pool became the bottleneck.
| Model and workload | Separate pools | Unified memory |
|---|---|---|
| Qwen3.5-4B, 1,024 input tokens, 256 output tokens, concurrency 64 | 5,880 output tokens/s | 5,830 output tokens/s with identical GSM8K accuracy |
| GPT-OSS-20B under maximum concurrency | 618 simultaneous requests; 43 s time to first token | About 1,000 simultaneous requests; 6 s time to first token; 14% higher throughput |
| Qwen3.5-4B, 16K input tokens, 4,096 output tokens, concurrency 64 | 22 active requests | 32 active requests and 26% higher throughput |
| Qwen3.5-4B, 512 input tokens, 16,384 output tokens, concurrency 51 | About 294 retractions per 1,000 completed requests | About 233 retractions per 1,000 completed requests |
A separate GPT-OSS-20B sensitivity test showed the cost of a poor static split: setting --swa-full-tokens-ratio to 0.1 reduced throughput by 12% relative to the default ratio.
A retraction occurs when memory pressure forces the scheduler to remove an in-flight request from the active batch. Lower retraction counts reduce scheduling churn and repeated work during long generations.
These figures cover one H200 and the listed workloads. Multi-GPU deployments and production traffic still require workload-specific validation.
Support across the serving stack
| Area | Supported paths |
|---|---|
| Scheduling and caching | Overlap scheduling, Radix cache, and HiCache |
| Distributed execution | Prefill/decode disaggregation, pipeline parallelism, decode context parallelism, and data-parallel attention |
| Speculative decoding | EAGLE, EAGLE3, DFLASH, and DSPARK; compatible fused drafts can share entry storage with target-model KV |
SGLang's usage guide documents configuration, while the design guide provides the backend compatibility matrix and allocator details.
Current limits shape deployment
- Unified memory remains experimental and targets hybrid models.
- Models containing only full-attention layers, MLA paired with sliding-window layers, and DeepSeek-V4 are unsupported.
- Prefill/decode disaggregation requires Mooncake and cannot run alongside pipeline parallelism, speculative decoding, or decode context parallelism.
- HiCache cannot run alongside EAGLE or EAGLE3.
- Priority preemption is disabled while unified memory is active.
Replay the traffic mix before rollout
Production evaluation should begin in staging with --enable-unified-memory and traces covering long-context, short-request, and high-concurrency traffic. Teams can compare output throughput, time to first token, maximum active requests, and retractions against their current split-pool configuration, then verify every enabled backend against SGLang's compatibility matrix.
Comments
