vLLM's TileRT Integration Hits 618 tok/s by Splitting AI's Two Hardest Jobs

LLM inference has a dirty secret: the two phases of generation, prefill (processing your prompt) and decode (generating tokens one by one), have fundamentally different hardware needs. Prefill is compute-bound; decode is memory-bandwidth-bound. Running them in the same engine forces an uncomfortable compromise. vLLM's new integration with TileRT is the clearest demonstration yet of what becomes possible once you stop making that compromise.

Today, vLLM is introducing vLLM prefill paired with TileRT decode, integrated through vLLM V1's public connector interface and shipping with TileRT 0.1.5. The headline is simple: you can now route latency-sensitive traffic to a purpose-built decode engine while keeping everything else, your API surface, prefix caching, scheduling, and tooling, exactly as it was.

Why disaggregation unlocks a new design space

Disaggregated serving, which separates the compute-bound prefill phase from the memory-bandwidth-bound decode phase, has become an increasingly standard pattern for serving large language models at scale. That architectural shift carries a benefit that is easy to overlook: once prefill and decode are separated, the decode side becomes pluggable.

PD disaggregation has rapidly become the industry standard, supported by all major serving frameworks including vLLM, SGLang, TensorRT-LLM, LMDeploy, and NVIDIA Dynamo, and deployed at production scale by providers such as DeepSeek and Gemini. But until now, "pluggable" was mostly theoretical. Every framework assumed you'd decode with the same engine that did the prefill. This integration makes the plug-in architecture real.

vLLM's native decode is, and remains, the right default: it is built for high-throughput batched serving across a huge range of models and hardware. But there is a growing class of workloads, such as agentic loops, interactive coding assistants, and real-time voice, where the metric that matters is not aggregate throughput but how fast tokens reach each individual user.

What TileRT actually is

TileRT serves large language models in ultra-low-latency scenarios, pushing the latency limits of hundred-billion-parameter models to millisecond-level time per output token without compromising model size or quality. Its tile-level runtime engine decomposes LLM operators into fine-grained tile tasks and dynamically overlaps computation, I/O, and communication across multiple GPUs.