vLLM's AFD Plugin Cuts DeepSeek-V3.2 Response Time by 47%
vLLM's new AFD Plugin splits Attention and expert computation into independent services, unlocking +11% decode throughput for large MoE models like DeepSeek-V3.2
- New plugin: vLLM AFD Plugin splits MoE model inference into separate Attention and FFN services, enabling independent scaling of each.
- Performance: A 64A16F layout delivers +11.3% decode throughput per die at 16K context and +9.0% at 32K vs. a standard EP64 baseline on Ascend 910C.
- Prefill gains: Async CAM connector cuts median time-to-first-token by ~47% (15.1s to 8.0s) at 12 req/s.
- Caveat: The attention-to-FFN ratio is critical -- a 48A16F layout actually underperforms EP64; disaggregation alone is not enough.
- Availability: Open-source under vllm-project/afd-plugin, targeting vLLM 0.19.1; supports NVIDIA GPU and Ascend NPU backends.
- Model support: Currently covers DeepSeek V2/V3 family (including V3.2) and GLM MoE DSA; broader model and hardware coverage on the roadmap.
Serving a Mixture-of-Experts (MoE) model like DeepSeek-V3.2 is two fundamentally different jobs running on the same hardware. Attention layers are memory-bandwidth-bound and tightly coupled to request scheduling and the KV cache. Expert FFN layers are compute-bound and dominated by token routing across many GPUs. A single homogeneous deployment forces a permanent tradeoff: optimize for one, and you're wasting resources on the other. The vLLM AFD Plugin resolves this by physically separating the two into independent, scalable services.
The plugin is available at vllm-project/afd-plugin, contributed by teams from Ascend, vLLM, StepFun, Ant Group, and FastAFD. It targets vLLM 0.19.1, requires Python 3.10–3.13, and integrates through vLLM's standard plugin entry point and --additional-config channel without forking the vLLM source tree.
Why the hardware mismatch gets worse at scale
Attention-FFN Disaggregation (AFD) assigns attention and expert/FFN computation to physically separate GPU groups. The mismatch between memory-bound attention and compute-bound expert MLPs compounds as you scale expert parallelism (EP): more GPUs thrown at experts means more waste on attention, and vice versa. A homogeneous cluster cannot be optimal for both simultaneously.
Production systems have already shipped this architecture. ByteDance's MegaScale-Infer demonstrated disaggregated expert parallelism at scale. StepFun's Step-3 used model-system co-design for cost-effective decoding. Huawei's xDeepServe deployed it across CloudMatrix384 as a model-as-a-service. The vLLM AFD Plugin brings this pattern into the open-source ecosystem as a pluggable extension to the most widely used open-source inference engine.