vLLM's TileRT Integration Hits 618 tok/s by Splitting AI's Two Hardest Jobs

LLM inference has a dirty secret: the two phases of generation, prefill (processing your prompt) and decode (generating tokens one by one), have fundamentally different hardware needs. Prefill is compute-bound; decode is memory-bandwidth-bound. Running them in the same engine forces an uncomfortable compromise. vLLM's new integration with TileRT is the clearest demonstration yet of what becomes possible once you stop making that compromise.

Today, vLLM is introducing vLLM prefill paired with TileRT decode, integrated through vLLM V1's public connector interface and shipping with TileRT 0.1.5. The headline is simple: you can now route latency-sensitive traffic to a purpose-built decode engine while keeping everything else, your API surface, prefix caching, scheduling, and tooling, exactly as it was.

Why disaggregation unlocks a new design space

Disaggregated serving, which separates the compute-bound prefill phase from the memory-bandwidth-bound decode phase, has become an increasingly standard pattern for serving large language models at scale. That architectural shift carries a benefit that is easy to overlook: once prefill and decode are separated, the decode side becomes pluggable.

PD disaggregation has rapidly become the industry standard, supported by all major serving frameworks including vLLM, SGLang, TensorRT-LLM, LMDeploy, and NVIDIA Dynamo, and deployed at production scale by providers such as DeepSeek and Gemini. But until now, "pluggable" was mostly theoretical. Every framework assumed you'd decode with the same engine that did the prefill. This integration makes the plug-in architecture real.

vLLM's native decode is, and remains, the right default: it is built for high-throughput batched serving across a huge range of models and hardware. But there is a growing class of workloads, such as agentic loops, interactive coding assistants, and real-time voice, where the metric that matters is not aggregate throughput but how fast tokens reach each individual user.

What TileRT actually is

TileRT serves large language models in ultra-low-latency scenarios, pushing the latency limits of hundred-billion-parameter models to millisecond-level time per output token without compromising model size or quality. Its tile-level runtime engine decomposes LLM operators into fine-grained tile tasks and dynamically overlaps computation, I/O, and communication across multiple GPUs.

The key technique powering TileRT's speed is MTP, or Multi-Token Prediction. Instead of generating one token per forward pass through the model, GLM-5's approach is to train the model to predict the next 2 to 3 tokens simultaneously, and then the main model quickly verifies which ones are acceptable. This is a form of speculative decoding built directly into the model architecture, not bolted on afterward.

An 8×H200 NVL server provides nearly 38 TB/s of aggregate memory bandwidth. For GLM-5.1, the activated parameter footprint during decode is only around 42 GB per token. From a purely theoretical bandwidth perspective, decode throughput could approach 1000 tok/s even without MTP enabled. Yet real systems often deliver only a few dozen tok/s. TileRT's entire design philosophy is about closing that gap.

Zero-fork integration: how it actually works

The core design principle is zero changes to vLLM: no fork, no patches, no wrapped internal workers. The integration lives entirely behind vLLM V1's public extensibility surface: a KVConnectorBase_V1 implementation, composed under MultiConnector and loaded through the standard kv_connector_module_path mechanism.

The routing logic is elegant in its simplicity. A lightweight router fronts the TileRT pool. For each request it sets max_tokens=1 (vLLM performs the prefill and emits the first token) and attaches the target decode node in the standard pass-through field: kv_transfer_params = {"tilert_host": ..., "tilert_ctrl_port": ...}. Traffic for the native pool flows through the usual disaggregation proxy, unmodified.

The KV handoff between engines is where the engineering gets interesting:

• RDMA one-sided writes: After prefill, the request's attention state moves to the decode node as RDMA one-sided writes into pre-registered GPU buffers, with either Mooncake or NIXL as the transfer engine. No intermediate serialization, no staging through host memory.
• Fully overlapped with prefill: State extraction happens inside the forward window: the request's state is copied to a staging buffer before its cache blocks can be recycled, and a background sender performs the actual network transfer. A request bound for TileRT never blocks the next prefill iteration.
• Live injection: On arrival, the state is converted to TileRT's native layout and injected directly into a running engine; decoding begins immediately, with multi-token speculative decoding active from the first step.

The TileRT connector claims only requests carrying the mark and is a strict no-op for everything else, so the two decode pools can share one prefill instance. This means adopting TileRT for your agent traffic has zero impact on your batch inference workloads running in the same deployment.

The benchmark numbers

GLM-5.1-FP8 token generation speed with TileRT v0.1.5 was measured with output length 1K and input length 1K to 192K. The bars compare TileRT without MTP, with MTP at average acceptance length 3.2, and the peak under best-case MTP acceptance.

The result: roughly 618 tok/s single-user decode on GLM-5.1-FP8 running on 8x B200s, approximately 2x the no-MTP baseline, with peaks near 800 tok/s under best-case MTP acceptance. For context, MTP enabled in TileRT reaches up to 600+ user TPS for DeepSeek-V3.2 as well, showing the pattern generalizes across frontier models.

When to use which pool

The integration is designed around a clear split:

• Route to TileRT decode when per-user token speed is the binding constraint: interactive agents, real-time voice assistants, latency-SLO inference, and the model is one TileRT supports.
• Stay on native vLLM decode for maximum aggregate throughput, high-concurrency batching, and the long tail of models and features that general-purpose decode covers.

Both stacks expose the same OpenAI-compatible surface, so moving a workload between them is a routing change, not a client change. That's the key practical win: no SDK changes, no prompt reformatting, no client-side migration.

There are real limitations to know about. In this release a TileRT decode node serves one in-flight request at a time, with the router providing gated dispatch and back-pressure. Model coverage in this release is GLM-5/5.1 and DeepSeek-V3.2, with more to come. The single-request-per-node constraint means you need to scale out decode nodes horizontally to serve concurrent users, which is a meaningful infrastructure consideration.

Getting started

TileRT 0.1.5 is available on PyPI (pip install tilert; Python 3.12, CUDA 13 wheels) and the TileRT repository. Install it on both the prefill and decode nodes; the prefill side needs it for the connector plugin.

The setup involves three components running in sequence:

1. Convert the HuggingFace checkpoint to TileRT's weight format using tilert.models.preprocess.weight_converter
2. Launch the TileRT decode server with python -m tilert.pd_vllm.decode_server
3. Start a stock vllm serve instance with the --kv-transfer-config flag pointing at the TileRT connector plugin
4. Run the PD router (tilert.pd_vllm.pd_router) as the OpenAI-compatible ingress

To run both TileRT and native vLLM decode pools simultaneously behind one prefill instance, you compose both connectors under MultiConnector. MultiConnector is a wrapper for using multiple KVConnectors at the same time. The validated configuration uses NIXL end-to-end for both pools, so only the prefill's --kv-transfer-config changes.

The bigger picture

As inference penetrates deeply into real-world applications, another law is becoming undeniably clear: speed itself is redefining the boundaries of model capability. This integration is a concrete proof point for a broader architectural shift: inference stacks are becoming compositions of specialized engines behind shared serving layers, not monolithic runtimes.

The assumption worth updating is that your serving framework and your inference engine have to be the same thing. vLLM's connector interface separates those concerns cleanly. The prefill pool, the scheduler, the API layer, and the caching infrastructure stay in vLLM. The decode engine becomes a deliberate, workload-specific choice. That's a meaningful expansion of the design space for anyone building latency-sensitive LLM applications.