vLLM v0.26.0 Ships Day-0 Support for Inkling's 1T-Parameter Multimodal Model

vLLM v0.26.0 is a dense release. This release features 411 commits from 212 contributors, including 61 first-timers. That contributor count is a signal worth paying attention to: vLLM is no longer just an NVIDIA project. AMD, Intel, IBM, and a growing list of model labs are all pushing code in. The result is a release that touches every layer of the stack, from low-level CUDA kernels to the Rust frontend and the OpenAI-compatible API.

The headline: a 1T-parameter model lands with Day-0 support

The most newsworthy addition is full support for the Inkling model family from Thinking Machines Lab. TML Inkling is a 1T-parameter multimodal model that natively accepts text, image, and audio inputs and generates text with up to 1M context length. It introduces several novel architecture components , relative attention, short convolution, and shared expert sinks , all of which are now efficiently integrated into vLLM.

The architecture is worth understanding before you try to deploy it. The model uses 66 layers, including 11 full-attention layers and 55 sliding-window attention layers. This heavy reliance on sliding-window attention is central to making million-token context practical. Instead of RoPE, Inkling uses relative attention, adding a learned relative-position term to attention logits. Each layer also applies a short convolution with window size 4 to attention keys, values, output, and MoE output.

On the serving side, Inkling includes eight MTP heads for speculative decoding , a technique where the model drafts multiple tokens in one forward pass and verifies them together , allowing up to nine tokens per step. vLLM reports up to 380 tokens per second per user with MTP on four NVIDIA GB200 GPUs, versus 140 tokens per second without MTP. vLLM also provides full feature parity, including LoRA, TP/DP/EP/PP parallelism, prefix caching, and disaggregated serving.

• Hardware target: Runs on both Hopper (H200 and H20) and Blackwell for BF16; NVFP4 variant targets NVIDIA Blackwell (B200/GB200).
• Scale: 975B total parameters, 41B active per forward pass, 1M-token context window.
• Quantization: Both thinkingmachines/Inkling-NVFP4 and the BF16 thinkingmachines/Inkling model are supported with optimized performance and full feature parity.
• Parallelism needed: 1T parameters requires multi-GPU. The featured configuration is 4x GB200 for the NVFP4 variant; the BF16 variant needs multi-node.

Flexible attention: the infrastructure change that matters most

Under the hood, the most architecturally significant change in v0.26.0 is how vLLM handles attention backends. Previously, a model used one attention backend for all its layers. Now, the attention backend can be selected per KV-cache group, and sliding-window support is now an explicit backend capability , improving support for hybrid models.

Why does this matter? Some models interleave more than one attention type across their layers , most commonly sliding-window attention on most layers and full attention on a few. vLLM serves these with its hybrid KV cache manager, which splits the model's layers into multiple KV cache groups, one per attention behavior. Before this change, memory was wasted because the engine had to treat sliding-window layers as if they were full-attention layers. The memory waste could be as high as 79.6% in some models, 25% in Gemma-2, and 56.25% in Ministral. With per-group backend selection, each group of layers gets the right kernel for its attention pattern, reducing that waste and enabling more efficient serving of the growing class of hybrid-architecture models like Inkling, Gemma-3, and Qwen3.

KV cache goes three-tier

KV cache , the stored key-value pairs from the attention mechanism that let the model avoid recomputing context on every token , has always been the main bottleneck for serving long contexts at scale. v0.26.0 takes a meaningful step toward solving this with a proper tiered storage system.

To date, vLLM offered native KV offloading to CPU memory but did not support further offloading from CPU memory to other tiers such as storage. v0.26.0 changes that. The new object-store secondary tier lets you push KV blocks from CPU DRAM into object storage, with workload identity for cloud auth, DP-replica-aware tiering so different data-parallel replicas don't clobber each other's cache, and new metrics for monitoring offload latency and lookup delays.

The practical hierarchy now looks like this:

• GPU HBM: Hot blocks actively being used for generation
• CPU DRAM: Warm blocks, completed but likely to be reused
• Object store: Cold blocks, historical context or prefix cache hits for future requests

Storage offloading increases effective KV-cache capacity, enables cross-replica reuse, and makes clusters more elastic by allowing them to scale efficiently with growing request volume and user concurrency. For RAG workloads with long shared system prompts, or multi-turn chat with many concurrent users, this is the difference between paying for more GPUs and just using the storage you already have.

DeepSeek-V4 gets a serious performance pass

DeepSeek-V4 continues to be the model that drives the most kernel work in vLLM. This release adds three targeted optimizations that compound on each other:

• A specialized routing kernel delivering 2.94% end-to-end TPOT (time per output token) improvement
• A fused_topk_bias kernel that runs 1.5 to 2x faster
• Redundant repeat/copy removal contributing another 1.8% E2E TPOT improvement

TPOT (time per output token) is the latency per generated token during the decode phase , the number that most directly affects user-perceived responsiveness in streaming applications. These gains are additive and apply across NVIDIA, ROCm, and XPU backends. AMD users also get a two-stage compressor kernel for HCA (high-concurrency attention) prefill, plus DSpark speculative decoding on ROCm and XPU.

What else is in the box

Quantization: The release adds Humming w[2-7]a[4,8] weight-only inference , a new quantization scheme supporting weights compressed to 2 through 7 bits with 4- or 8-bit activations. There's also nvfp4_per_token online MoE quantization and INT2 weight-only inference on XPU. For generation accuracy, fp32 lm_head for generation models is now available via head_dtype, extended to the LoRA path and given a ROCm fast path, improving accuracy for generation heads.

Security: Several important fixes landed. The diskcache dependency was replaced to eliminate pickle deserialization , a common vector for arbitrary code execution in Python services. A concurrent sparse-invariant race that bypassed a previous CVE fix was also patched, along with bounded prompt lists and a regex-compilation timeout to prevent denial-of-service via structured output constraints.

Rust frontend: The Rust frontend gained multimodal video and audio support, a Seed-OSS tool parser, and a native vllm-bench port. The Rust frontend is vLLM's high-performance request router and tokenization layer , adding video and audio there means multimodal workloads can benefit from its lower overhead compared to the Python path.

New models beyond Inkling: BertForMaskedLM, RobertaForTokenClassification, XLMRobertaForTokenClassification, Cosmos3 Edge Reasoner, and TranslateGemma-12b-it all land in this release, broadening vLLM's reach into encoder-only and specialized translation workloads.

CPU and edge: Native macOS arm64 CPU wheel builds are now available, alongside s390x NUMA topology support and IBM POWER VSX math optimizations. vLLM is clearly not just a cloud-GPU framework anymore.

Before you upgrade

There are breaking changes to plan for:

• Dependency bumps: Transformers 5.13.0, FlashInfer 0.6.14, and NIXL 1.3.1 are all required. Check your environment before upgrading.
• Models removed: TeleChat, Persimmon, and Fuyu have been removed. If you're serving any of these, stay on v0.25.x.
• ROCm version: vLLM v0.21.0 and later uses ROCm 7.2.2.
• Python/OS requirements: Python 3.12 and glibc >= 2.35 (Ubuntu 22.04+) are required.

Install with:

pip install vllm==0.26.0
# or with uv for the latest
uv pip install vllm --torch-backend auto

The release is available now on PyPI. The combination of Inkling Day-0 support, per-group attention backends, and tiered KV storage makes v0.26.0 one of the more architecturally significant vLLM releases in recent memory , especially for teams running hybrid-architecture models or pushing the limits of context length at scale.