Ant Group Pushes Qwen3-Omni to 5.4x Faster With 0.6s Audio Response

Serving a model that listens, reasons, and speaks in real time turns out to be a very different problem from serving a text LLM. The vLLM-Omni team and Ant Group's Super Computing Technology team just published a detailed breakdown of how they got Qwen3-Omni production-ready , and the numbers are striking: first audio in ~0.6 seconds instead of ~6, speech generated faster than real time, and 5.4x more throughput on the same hardware.

Three models pretending to be one

Qwen3-Omni is Alibaba Qwen's fully omnimodal model. It adopts a Thinker-Talker Mixture-of-Experts architecture that unifies perception and generation across text, images, audio, and video. Across 36 audio and audio-visual benchmarks, Qwen3-Omni achieves open-source state-of-the-art on 32 benchmarks and overall SOTA on 22, outperforming strong closed-source models such as Gemini-2.5-Pro, Seed-ASR, and GPT-4o-Transcribe.

Under the hood, it runs as three distinct stages with very different compute profiles:

• Thinker , the heavy multimodal reasoning engine. It ingests text, images, audio, and video, then produces text tokens and hidden states (rich internal representations of meaning).
• Talker , receives those hidden states and autoregressively generates discrete speech codec codes (compressed audio tokens) frame by frame. To achieve ultra-low-latency streaming, Talker autoregressively predicts a multi-codebook sequence. At each decoding step, an MTP module outputs the residual codebooks for the current frame, after which the Code2Wav renderer incrementally synthesizes the corresponding waveform, enabling frame-by-frame streaming generation.
• Code2Wav , a neural vocoder that converts the codec codes into actual audio waveforms.

The core serving challenge: these three stages hit completely different bottlenecks. Treating them as one loop forces the slowest sub-path to gate everything else.

The optimization stack, layer by layer

The team validated each optimization on top of the previous one, benchmarking with 10 to 640 prompts at concurrency 1 to 64 on Qwen3-Omni-30B-A3B-Instruct. Here's what each layer contributed at concurrency 64:

RTF (Real-Time Factor) is the ratio of audio generation time to audio playback time. An RTF below 1.0 means the system generates audio faster than it plays , the baseline at 1.15 was literally too slow to keep up with playback under load. The full stack brings it to 0.47.

What each optimization actually does

Stage decomposition and batching is the prerequisite for everything else. By treating each stage as an independent runtime with its own scheduler, batching policy, and graph capture, the team broke the coupling that forced one policy on all three stages. Collecting concurrent requests into a single Talker forward pass also closed the GPU idle-SM gap that single-request micro-work left open.

CUDA Graph was the single biggest win: a nearly 4x throughput jump. The idea is to record the sequence of GPU operations once at warmup and replay it with minimal CPU involvement on every subsequent step , eliminating the repeated Python-side kernel dispatch that dominated latency. Each stage gets its own capture strategy:

• Thinker uses vLLM's standard outer decode graph.
• Talker uses the outer graph plus torch.compile on its inner codec predictor (a 5-layer transformer that emits residual codec codes at each step).
• Code2Wav uses an inner CUDAGraphDecoderWrapper that pre-captures the vocoder's stable (batch, quantizers, frames) shapes.

Async chunk delivered the largest single drop in first-audio latency , from 2790ms to 655ms. Before this, the pipeline was barrier-synchronized: Talker couldn't start until Thinker finished its full generation, and Code2Wav couldn't emit audio until Talker had accumulated a complete payload. Async chunk replaces those barriers with pipelined partial handoffs. Thinker emits embedding rows incrementally, Talker slices on chunk boundaries, and Code2Wav starts producing audio after just a few codec frames , not after the entire Thinker generation.

Async output decouples payload construction from the decode loop. Previously, Thinker had to fully assemble each connector payload (copying embeddings and hidden states) before the next decode step could start , a synchronous gap that wasted GPU time. Moving that work off the hot path shrinks the inter-step gap from ~2.8ms to ~41 microseconds.

Stage replicas address an asymmetry in the pipeline: for each request, Thinker generates text once, but Talker and Code2Wav then run hundreds of short steps to render that text as audio. Under load, the speech stages saturate first while the heavy Thinker still has headroom. The solution is to replicate only the bottleneck stages , keeping one Thinker on GPU 0 and running 2x Talker + 2x Code2Wav on GPUs 1 and 2:

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
  --stage-overrides '{"1": {"num_replicas": 2, "devices": "1,2"}, \
                      "2": {"num_replicas": 2, "devices": "1,2"}}'

Hot-path cleanup tackled model-internal overhead that scales with utterance length: redundant connector traffic, per-step torch.cat allocations, and unnecessary device-to-host tensor copies. After cleanup, a long-context single-request test saw end-to-end latency drop from 21.28s to 7.37s, and audio RTF fall from 0.71 to 0.28.

Getting started

The stack is available now in vLLM-Omni, the vLLM project's dedicated omnimodal serving framework. The 0.16.0 release rebases onto upstream vLLM v0.16.0 and significantly expands performance, distributed execution, and production readiness across Qwen3-Omni, with platform coverage across CUDA, ROCm, NPU, and XPU. The simplest deployment resolves the full staged profile automatically:

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \
  --omni \
  --port 8091

Requests go to /v1/chat/completions with a modalities field in the body , set to ["text"] for text-only or ["text", "audio"] for speech output. The deploy config auto-detects the runtime backend, so the same command works across GPU vendors without extra flags.

Why this matters beyond Qwen3-Omni

The real contribution here isn't a single trick , it's a framework for thinking about multi-stage generative pipelines. Most multimodal models that produce audio or video will have this same structure: a heavy reasoning stage followed by lighter but high-frequency generation stages. The lesson from this work is that each stage needs its own runtime policy, and the handoffs between stages are just as important to optimize as the stages themselves.

Qwen3-Omni is a single multimodal model that, for the first time, maintains state-of-the-art performance across text, image, audio, and video without any degradation relative to single-modal counterparts. It matches the performance of same-sized single-modal models within the Qwen series and excels particularly on audio tasks. Getting that model to run at real-time speeds under concurrent load required treating it not as one model, but as a pipeline , and optimizing every joint in that pipeline independently. That's a pattern the field will need to repeat as omnimodal models become the norm.