vLLM v0.28.0 Ships Sparse Attention and 60% Faster Speculative Decoding

The vLLM team just cut v0.28.0, and it is one of the meatier releases in recent memory. The changelog lists 584 commits from 270 contributors, 76 of them first-timers, and covers most of what the inference stack has been chasing this cycle: sparse attention for frontier MoE models, smarter speculative decoding, tiered KV cache offloading with a disk tier, and a much more mature Model Runner V2.

If you serve open-weight models at any real scale, read this one carefully before bumping the version pin. Several breaking defaults will bite you if you skip the notes.

Frontier model support gets serious

The headline effort is a stack-wide push around Kimi-K3. That includes Decode Context Parallel (DCP) support, fused FlashKDA decode and prefill kernels, SiTU activation support for MegaMoE, GEMM-RS for sequence parallelism, combined all-gathers with a 1.5 to 3x kernel-level speedup, an adaptive speculative token budget delivering roughly 60% better DSpark TTFT, and optional shared-expert sharding that saves about 17 GiB of memory per GPU. The K3 path also lights up on AMD ROCm through the V2 model runner.

DeepSeek-V4 is the other major target. Sparse MLA now works end-to-end for plain decode, MTP, and DSpark speculative decoding, joined by AMD Quark NVFP4 support, reasoning-effort prompts and mappings, sparse top-k metadata kernel optimizations, narrowed eager CUDA graph regions, and ROCm enablement on gfx11 and gfx950. If you were waiting for a production-grade V4 serving story on either vendor, this release closes most of the gaps.

Several new architectures also land: Muse Glimmer, Ling 3.0 Flash with BF16, MTP, and parser support plus an FP8 variant and hybrid MXFP4 routed experts, Dots3 NOTE native multimodal support, and Interns2mobius.

Speculative decoding grows up

Speculative decoding is where a lot of the real latency wins live now, and v0.28.0 pushes hard on it. DFlash2 arrives with local convolution and a candidate selector, DSpark gains confidence-scheduled verification, and async scheduling is auto-enabled for draft models.

The most interesting knob is the adaptive speculative token budget. Rather than committing to a fixed number of drafted tokens per step, the scheduler adjusts on the fly based on how much verification the accepted draft actually deserves. The team reports a 55 to 65% improvement in end-to-end time-to-first-token on DSpark workloads, a large jump for what amounts to a scheduling tweak.

Model Runner V2 absorbs the last holdouts

Model Runner V2 keeps pulling in features that used to be V1-only. This release adds E/P/D disaggregation, weight offloading, multi-layer MTP KV cache support, encoder CUDA graphs, decoder token-wise pooling plus Transformers pooling models, attention-free models, and thinking_token_budget support. Most non-exotic serving setups can now sit on V2 without missing critical features.

The KV cache story also gets more interesting. Tiered offloading adds disk offloading support, out-of-tree secondary tier managers via module_path, partial secondary-tier load results, tiering metrics, and a canonical CPU layout for parallelism-agnostic offload. A disk tier means you can cache very long shared prefixes (system prompts, retrieved documents, agent scratchpads) across restarts, or across a much larger working set than fits in DRAM.

Four backends, one release

The hardware section is genuinely cross-vendor this cycle. Highlights per backend:

• NVIDIA: FlashInfer XQA decode support on SM12x, a CuTeDSL fused query kernel on SM100, programmatic dependent launch for DSA decode kernels, the native DSA decode path for MTP=3 on SM90, GB10 fused-MoE FP8 tuning configs, and B12X dense linear backends.
• AMD ROCm: torch 2.12 and triton 3.7 stack bump, AITER and FP8 inference on GFX120x, DeepSeek-V4 on gfx11, optimized Triton sparse-MLA decode on gfx950, a FlyDSL decode-attention kernel for 4-bit TurboQuant KV cache, and a fused Kimi-K3 KDA decode kernel.
• Intel XPU: a torch linear backend including blockwise GEMM, MXFP8 linear weights for the INC DeepSeek V4 model, async-scheduling PP sampled-token broadcast overlapped with compute, and an XPU wheel added to the release pipeline.
• CPU: an MLA backend so DeepSeek-V2/V3 can run on CPU, a triton-cpu wheel, GPTQ and AWQ enabled on s390x, and BF16 MoE routed through zentorch on AMD.

Quantization, serving, and security

On the quantization side, online MXFP4 support arrives, online weight scales are shared across TP, precision is preserved in online NVFP4 expert packing, and the online NVFP4 MoE kernel is reused across reloads. There is also batch-invariant NVFP4 MoE via CUTLASS, which matters if you have been chasing reproducibility issues between batch sizes.

On the frontend, request priority can now be parsed from an HTTP header, session ID plumbing lands in requests, and the streaming parser exposes count_reasoning_tokens. The Rust frontend keeps expanding with a standalone renderer, gRPC multimodal image inference, and explicit data-parallel rank routing.

A DoS via sample-rate forgery that bypassed the audio decode duration guard was fixed, and the audio decode duration limit is now also enforced in NanoNemotronVL. If you serve any multimodal endpoint that accepts audio, patch.

Read this before you upgrade

A few defaults and removals will surprise you if you just bump the version:

1. max_num_batched_tokens is raised from 8192 to 16384, prefix caching is enabled by default for Mamba models, and the Blackwell CUDA graph capture default is raised to 1024.
2. bitsandbytes support is now an out-of-tree plugin. If your deployment relies on it, install the plugin or your model load will fail.
3. The deprecated calculate_kv_scales runtime KV scale calculation was removed, and override_attention_dtype was removed.
4. Transformers is bumped to 5.15.0, and the runtime image is upgraded to Ubuntu 24.04.
5. KV offload tiering metrics are renamed from kv_offload_tiering_block_{queries,hits} to the chunk equivalents. Update your Prometheus dashboards.

Install is unchanged for most people: pip install vllm pulls the CUDA 13.0 wheel by default, with CUDA 12.9, ROCm, CPU, and XPU images all published under the vllm/vllm-openai tag family. Given how much of this release is about squeezing more tokens per second out of the same hardware, teams running MoE or reasoning models will want to test it as soon as their pinned defaults are sorted out.