SGLang adds 4-bit NVFP4 KV caching for Blackwell GPUs

Long-context inference increasingly runs into GPU memory limits because every prompt token, tool trace, and generated token adds key and value vectors to the attention cache. This KV cache prevents the model from recalculating prior tokens during generation. When it fills, SGLang must evict reusable prefixes and recompute them if a later turn needs that context.

SGLang, Qwen, and NVIDIA have added experimental NVFP4 KV caching on Blackwell GPUs, according to their joint announcement. The implementation reduces KV storage to about 56% of the equivalent FP8 footprint, providing a nominal 1.78× increase in cache capacity and 26% to 30% higher decode throughput at matched concurrency in the reported tests.

The feature is available through one runtime option:

--kv-cache-dtype nvfp4

Nine bytes replace 16

FP8 KV storage already uses roughly half the memory of BF16, but additional compression increases quantization error. NVFP4 controls that error with two scaling levels: each block of 16 four-bit E2M1 values receives an FP8 scale, and the tensor receives an additional FP32 scale. E2M1 allocates two exponent bits and one mantissa bit to each stored value.

Each 16-value block occupies eight bytes of packed data plus one byte for its block scale. The same values require 16 bytes in FP8, so the NVFP4 block consumes 56.25% as much space before alignment, allocator, and tensor-scale overhead. Dividing 16 by nine yields the nominal 1.78× capacity increase.

Three attention paths share one cache

SGLang handles the compressed cache differently during initial prompt processing, later context extension, and token generation:

1. Initial prefill: Attention runs with BF16 queries and FP8 key/value tensors. SGLang quantizes the key/value tensors to NVFP4 as it writes them into the persistent cache.
2. Chunked prefill and extend: The runtime gathers cached NVFP4 prefixes, dequantizes them into an FP8 workspace, and combines them with the current chunk. A single pair of workspace buffers is reused across layers to avoid separate allocations for every layer.
3. Decode: The attention kernel reads packed NVFP4 entries directly and dequantizes them during computation, avoiding a separate pass that would materialize the full cache in a wider format.

Decode receives the clearest performance benefit because each generated token reads the accumulated KV cache, making the operation sensitive to memory bandwidth. Fusing dequantization into attention reduces the bytes transferred from GPU memory on every decode step.

On the SM120 kernel path, the implementation loads packed key/value tiles and E4M3 block scales into shared memory. The cvt.rn.bf16x2.e2m1x2 instruction expands pairs of E2M1 values into BF16 registers before the attention matrix multiplication, without writing a fully dequantized cache back to GPU memory.

The largest reported gap is 1.6 points

Published accuracy tests covered Qwen3.5-397B-A17B and Qwen3.8-27B on GSM8K, GPQA-Diamond, AIME 2025, and SWE-bench Verified. Model weights remained in FP8, isolating the KV-cache data type as the tested variable.

The 397B mixture-of-experts model stayed closest to FP8 across the reported tasks. The 27B model showed larger variation, including a 1.60-point gap on SWE-bench Verified. These results cover two models and a limited benchmark set, so teams should validate quality on their own prompts, decoding settings, and agent workflows.

Longer contexts unlock larger batches

Throughput tests used Qwen3.8-27B on one RTX PRO 6000 Blackwell Server Edition GPU. Inputs contained 32K, 160K, or 1M tokens, and each request generated 1K output tokens.

With request concurrency held constant, NVFP4 improved peak-batch decode throughput by approximately 26% to 30% across the three context lengths. This comparison isolates much of the memory-bandwidth benefit because both formats serve the same number of resident requests.

Capacity-driven tests allowed each format to use its largest supported batch, combining the bandwidth reduction with NVFP4’s additional cache headroom:

The capacity-driven gains include two effects: less KV traffic per decode step and larger resident batches. They therefore exceed the improvement attributable to the fused decode kernel alone.

Prefill remains compute-bound, so compression provides little direct acceleration during initial prompt processing. At matched concurrency, mean time to first token increased by 0.20% to 0.40%, primarily from quantization overhead. Under capacity-driven load, the reported mean fell by as much as 10% because higher cache residency reduced queueing and recomputation.

Prefix residency drives the agent result

Multi-turn agents repeatedly alternate between adding tool results or user messages and decoding new output. Their latency depends heavily on whether earlier prefixes remain cached. An eviction can force the server to process hundreds of thousands of prior tokens again before generation resumes.

AgentX tests ran Qwen3.5-397B-A17B across eight RTX 6000D GPUs with eight-way tensor parallelism, reported as TP8. FP8 and NVFP4 behaved similarly at low concurrency. Above 12 concurrent requests, FP8 throughput and interactivity deteriorated as reusable prefixes were evicted, while NVFP4 throughput continued to rise. The corresponding cache-hit curves showed FP8 losing prefix residency earlier than NVFP4.

Support stops at a narrow boundary

The SGLang documentation labels NVFP4 KV caching experimental and defines several deployment constraints:

• GPU architecture: Support is limited to Blackwell SM12x, SM100, and SM103 devices. These SM identifiers refer to NVIDIA streaming-multiprocessor architecture targets.
• Attention layout: Current coverage includes grouped-query attention, or GQA, and sparse multi-head latent attention, or sparse MLA. Sparse GQA support remains in development.
• Scale calibration: The reported experiments fixed NVFP4’s tensor-level FP32 scale at 1.0. Calibrating that scale may reduce the remaining quality differences.
• End-to-end performance: Decode gains have less influence on workloads dominated by initial prefill, short generations, CPU-side orchestration, or tool latency.
• Practical capacity: The 1.78× figure follows from block storage. Model weights, activations, workspaces, allocator fragmentation, and runtime metadata reduce the increase in total request capacity.
• Benchmark scope: The published quality results cover two models, and the detailed throughput results use one model on one GPU configuration.

A production test needs two load profiles

Teams deploying on supported Blackwell hardware can evaluate NVFP4 without changing model weights by adding --kv-cache-dtype nvfp4 and comparing it with the existing FP8 configuration.

1. Run quality evaluations on representative prompts, tool traces, context lengths, and decoding parameters.
2. Measure decode throughput at matched concurrency to isolate the bandwidth benefit.
3. Repeat at each configuration’s maximum safe concurrency to measure the capacity benefit.
4. Track cache-hit rate, prefix evictions, time to first token, inter-token latency, and GPU memory usage.
5. Test sustained multi-turn sessions, where recomputation costs can dominate isolated-request benchmarks.

Long-context services with large reusable prefixes and memory-bound decode stand to gain the most from NVFP4. Short-prompt or prefill-heavy workloads will see smaller end-to-end changes even when cache capacity increases.

Smaller GPU-resident entries also affect hierarchical-cache tuning with systems such as HiCache. More prefixes can remain on the GPU before spilling to host memory, although transfer policy, non-KV allocations, and workload concurrency still determine effective session capacity.