vLLM Splits Multimodal AI Into Three Stages, Cutting Latency by 13%

vLLM has expanded EPD disaggregation so multimodal encoders can run, scale, and transfer outputs independently from language-model stages. The design addresses a common serving imbalance: images, video frames, and audio clips consume encoder capacity before generation begins, while prefill and decode impose different compute and memory demands.

Three stages, separate budgets

A multimodal request moves through three stages, each with a distinct workload:

EPD supports two deployment patterns:

• E+PD: A separate encoder sends embeddings to an instance that combines prefill and decode.
• E/P/D: Encoding, prefill, and decode each run as separate services.

Image-heavy workloads can add encoder replicas without adding decode capacity. Workloads with long responses can scale decode separately. This avoids provisioning language-model GPUs solely to absorb spikes in vision or audio processing.

Metadata cuts duplicate work

A basic disaggregated path sends media to the encoder, transfers the resulting embeddings, and forwards the original request downstream. The consumer may then fetch, decode, resize, normalize, or divide the same media into patches to reconstruct its input layout.

Metadata-only forwarding replaces the downstream media item with a reference containing structural information produced during encoding. For Qwen-family image inputs, the encoder publishes the computed image_grid_thw, which describes the temporal, height, and width grid. Prefill can use that value without reading the pixels. Video requests carry grids and timestamps, while audio requests carry token counts and feature lengths.

In a Qwen3-Omni benchmark, metadata-only forwarding raised throughput by 17.6% and reduced mean latency by 13.0%:

The encoder gets a lean runner

An encoder-only instance executes the media model and returns embeddings. It has no language model, key-value cache, or token sampler, so vLLM now gives that path dedicated loading and execution behavior:

• Model Runner V2 uses a dedicated MMEncoderModelRunner to manage GPU completion events without relying on token sampling.
• Producer-only encoder connector instances enter encoder-only mode automatically, and asynchronous execution steps omit sampling work.
• Encoder-only loading skips language-model checkpoint shards when vLLM can classify the weight layout safely. On a pruned Kimi-K3 checkpoint, the encoder read 2 of 96 shards, reducing weight-loading time from 683 seconds to 0.31 seconds.
• Vision Transformer CUDA Graph capture is available as an opt-in optimization for the encoder-only path.
• GPU preprocessing can use NVIDIA NVDEC hardware video decoding. On a 30-frame MMVU workload, end-to-end throughput improved by approximately 94% to 115% across three runs, each performed on a separate four-GPU GB200 node.

Embeddings cross machine boundaries

Cross-node EPD now supports network transports that move encoder outputs between machines:

• NIXL with a CPU encoder connector offloads outputs to registered, memory-mapped CPU regions and transfers them through NIXL.
• Mooncake encoder connector uses Mooncake TransferEngine over TCP or remote direct memory access.

The transfer protocol records the output width for each modality, meaning the number of values in each output vector. The tested Qwen3-Omni checkpoint produces visual outputs with a width of 8,192, including DeepStack information, and audio outputs with a width of 2,048. A single model-wide hidden-size field cannot represent both shapes.

Remote-transfer failures now return an unavailable request. This behavior prevents a process crash when the consumer holds only metadata and therefore cannot perform local encoding as a fallback.

One cache across encoder replicas

Instance-local caches lose reuse when routing sends repeated media to another encoder. Mooncake Store provides a shared cache across replicas: after a local miss, an encoder checks the Store, consumes any matching output, and publishes newly computed outputs asynchronously.

A controlled reassignment experiment used InternVL3_5-38B with two encoders and four combined prefill/decode instances across six H20 GPUs. Shared reuse produced the following results over 128 requests:

Time per output token remained essentially unchanged, indicating that the shared cache affected the encoding and prefill path rather than decode speed. The experiment deliberately reassigned every image, so its results represent an upper bound for workloads with less reuse across replicas.

GB200 gains appear under load

The DeepSeek V4.1 Flash benchmark compared combined prefill/decode deployment with EPD on 512 MP-DocVQA questions. Each request contained four to six document pages. The baseline used eight GPUs, while EPD added four single-GPU encoders for a total of 12 GPUs. Per-GPU throughput calculations included all 12 EPD GPUs.

Concurrency here means the number of requests being processed simultaneously. The crossover shows that EPD efficiency depends on sustained load, media mix, and replica allocation. Capacity planning should therefore use production-like request sizes and concurrency levels.

From two GPUs to three tiers

Three orchestration layers expose EPD through different deployment models:

vLLM v0.32.0 includes the core EPD capabilities, along with MiMo and XPU support. DeepSeek V4.1 Flash metadata-only support is available on the main branch.

A practical evaluation can begin with the encoder guide and an E+PD deployment on two GPUs:

1. Run one encoder instance and one combined prefill/decode instance.
2. Verify embedding transfer and metadata-only handling for every required media type.
3. Measure encoder utilization, throughput, latency, and time to first token at the target concurrency.
4. Add encoder replicas or separate prefill and decode after measurements identify the saturated stage.