vLLM's AFD Plugin Cuts DeepSeek-V3.2 Response Time by 47%

Serving a Mixture-of-Experts (MoE) model like DeepSeek-V3.2 is two fundamentally different jobs running on the same hardware. Attention layers are memory-bandwidth-bound and tightly coupled to request scheduling and the KV cache. Expert FFN layers are compute-bound and dominated by token routing across many GPUs. A single homogeneous deployment forces a permanent tradeoff: optimize for one, and you're wasting resources on the other. The vLLM AFD Plugin resolves this by physically separating the two into independent, scalable services.

The plugin is available at vllm-project/afd-plugin, contributed by teams from Ascend, vLLM, StepFun, Ant Group, and FastAFD. It targets vLLM 0.19.1, requires Python 3.10–3.13, and integrates through vLLM's standard plugin entry point and --additional-config channel without forking the vLLM source tree.

Why the hardware mismatch gets worse at scale

Attention-FFN Disaggregation (AFD) assigns attention and expert/FFN computation to physically separate GPU groups. The mismatch between memory-bound attention and compute-bound expert MLPs compounds as you scale expert parallelism (EP): more GPUs thrown at experts means more waste on attention, and vice versa. A homogeneous cluster cannot be optimal for both simultaneously.

Production systems have already shipped this architecture. ByteDance's MegaScale-Infer demonstrated disaggregated expert parallelism at scale. StepFun's Step-3 used model-system co-design for cost-effective decoding. Huawei's xDeepServe deployed it across CloudMatrix384 as a model-as-a-service. The vLLM AFD Plugin brings this pattern into the open-source ecosystem as a pluggable extension to the most widely used open-source inference engine.

How the plugin works

The architecture has three main components:

• Attention service: Retains vLLM's full scheduler, KV cache, batching, and sampling. It publishes batch metadata to the FFN side at each layer boundary.
• FFN service: A lightweight daemon with no scheduler or KV cache. It runs in a background loop, receives activations and routing metadata from the attention side, calls compute_ffn_output(), and returns results.
• Connector layer: A backend-neutral interface that handles tensor transfer between the two services. Each backend implements its own optimized communication path independently.

The split is invisible to clients. Requests still arrive at the same OpenAI-compatible endpoint, and the plugin integrates without touching the vLLM source tree.

Three connectors are currently supported:

Benchmarks: when disaggregation helps

The team benchmarked DeepSeek-V3.2 W8A8 on Ascend 910C hardware, comparing a standard EP64 deployment (64 GPUs handling everything) against two AFD layouts: 48A16F (48 attention ranks, 16 FFN ranks, 64 total dies) and 64A16F (64 attention ranks, 16 FFN ranks, 80 total dies).

At 16K fixed input length, EP64 achieves 232.6 tokens/s/die, 48A16F achieves 220.3 tokens/s/die (−5.3%), and 64A16F achieves 258.9 tokens/s/die (+11.3%). At 32K, EP64 reaches 168.2 tokens/s/die, 48A16F drops to 151.4 (−10.0%), and 64A16F hits 183.3 (+9.0%).

The ratio of attention to FFN ranks determines the outcome. 48A16F underperforms EP64 because it lacks enough attention capacity to saturate the FFN side. 64A16F, with more attention ranks, delivers the highest normalized throughput. Disaggregation alone does not guarantee a gain.

On prefill, the async connector shows a sharper improvement. At 12 requests per second, median time-to-first-token drops from 15.1 seconds to 8.0 seconds, a 47% reduction.

What heterogeneous hardware unlocks

ByteDance's MegaScale-Infer ran on heterogeneous clusters with H20 GPUs for attention and L40S GPUs for FFNs, achieving up to 3.24x higher throughput per dollar than baselines. That result points to a practical consequence of disaggregation: cheaper, memory-heavy GPUs can handle attention while compute-optimized GPUs handle experts. The vLLM AFD Plugin roadmap explicitly targets heterogeneous hardware support as a next step.

AFD also reframes how you think about scaling MoE deployments. The question shifts from "how many GPUs do I need for EP64?" to "what is the optimal attention-to-FFN ratio for this workload and sequence length?" The answer depends on context length, request rate, and hardware mix, which is exactly why the plugin exposes configurable layouts rather than a fixed topology.

Current limitations and what's coming

The plugin is experimental. Current limitations include:

• Pinned to vLLM 0.19.1 (model runner v1 only)
• Full model weights loaded on both attention and FFN roles (no weight sharding between them yet)
• Graph execution only in decode-only mode; the async prefill connector does not support graph capture yet
• Dual Batch Overlap (DBO) limited to exactly two micro-batches
• Only DeepSeek V2/V3-family and GLM MoE DSA model wrappers are currently registered

The roadmap includes broader vLLM version tracking, more flexible graph modes, production-scale validation, expanded model coverage, heterogeneous hardware placement, and potential integration with vLLM-Omni for multimodal pipelines.

If you're running large MoE models at scale, particularly DeepSeek-family models at long context lengths, the throughput ceiling on homogeneous EP deployments is real and AFD is a direct response to it. The repository includes deployment recipes to get started.