vLLM-Omni v0.22.0 Ships NVIDIA Cosmos 3 Support and Robot Serving

vLLM-Omni v0.22.0 just landed, and it is the most ambitious release the project has shipped. What started as an extension of vLLM to handle multimodal inputs has grown into a full production serving stack for world models, robots, speech synthesis, and video diffusion , all under one OpenAI-compatible API. With 339 commits from 124 contributors (52 of them brand new), this release signals that the community around omnimodal serving is growing fast.

The world model moment

The headline feature is day-0 support for NVIDIA Cosmos 3, announced at COMPUTEX 2026. Cosmos 3 is a leaderboard-topping open physical AI foundation model built on a mixture-of-transformers architecture, and it is the world's first fully open omnimodel with native vision reasoning and multimodal generation across text, image, video, ambient sound, and action. That last word , action , is what makes it different from every other multimodal model in the ecosystem.

NVIDIA trained Cosmos 3 on 20 trillion tokens of multimodal data, including nearly a billion images and 400 million real and synthetic videos. The action data is what makes Cosmos different from a regular video generator , it is meant to model how machines move, not just how scenes look. Previous Cosmos releases separated world generation, physical understanding, and controlled scene generation into different models and workflows. This release unifies those capabilities with a Mixture-of-Transformers architecture built around two towers: a reasoner tower that is a VLM interpreting multimodal observations, and a generation tower that creates physically grounded outputs.

vLLM-Omni v0.22.0 ships full Cosmos 3 support across model execution, recipes, tests, and accuracy coverage , including base model support, sound generation, and the action modality. You can now serve a model that takes in video of a robot arm, reasons about what it sees, and outputs both a predicted future video and the joint angles to execute the task, all through a single API endpoint.

Robots as first-class citizens

Beyond Cosmos 3, this release adds two robot-specific serving primitives that have no equivalent in the original vLLM:

• DreamZero + OpenPI realtime API. DreamZero is a 14B World Action Model that jointly predicts video and actions, enabling effective learning from diverse robot data. It achieves over 2x improvement in zero-shot generalization to unseen verbs and motions compared to state-of-the-art VLAs, with model and system optimizations achieving 38x inference speedup for real-time closed-loop control at 7Hz. vLLM-Omni now ships a dedicated realtime OpenPI robot serving API alongside DreamZero integration with CFG parallel (classifier-free guidance run across multiple parallel streams to improve output quality).
• veRL-Omni RL integration. Qwen3-Omni Thinker LoRA support for reinforcement learning training is now included, letting you fine-tune omnimodal models with RL directly against the serving stack.

Production TTS gets serious

Audio serving has been a weak point of most inference frameworks , they handle text well but treat speech as an afterthought. This release changes that. The TTS stack now covers Qwen3-TTS, Qwen3-Omni, VoxCPM2, Fish Speech S2 Pro, OmniVoice, GLM-TTS, Higgs Audio v2, MOSS-TTS, and Covo-Audio-Chat. More importantly, the serving infrastructure behind them has been hardened for production:

• Precomputed custom voices and a ref-context cache for Qwen3-TTS to avoid recomputing voice embeddings on every request
• Cross-request Code2Wav batching restored, which groups audio decoding work across concurrent users for higher throughput
• A bounded active-stream window to prevent memory blowup under high concurrency
• New audio SLO (service-level objective) metrics and audio streaming continuity metrics for Prometheus, so you can actually monitor TTS quality in production
• Async audio input support and voice clone serving for OmniVoice

For Qwen3-Omni specifically, a TTFP (time-to-first-packet) optimization using initial_codec_chunk_frames reduces the latency before the first audio chunk arrives at the client , critical for conversational applications.

Faster image and video generation

The diffusion stack (the part of vLLM-Omni that handles image and video generation models, which use a different architecture from text models) gets significant parallelism upgrades:

• Wan 2.2 now supports pipeline parallelism, splitting the model across GPUs in sequence rather than just replicating it
• HunyuanVideo 1.5 adds USP (Unified Sequence Parallelism) plus VAE patch parallelism, splitting both the transformer and the image decoder across GPUs
• LTX-2.3 adds CFG parallel, running the conditional and unconditional passes simultaneously on separate GPUs
• CacheDiT acceleration is now available for Helios, DreamID-Omni, SenseNova U1, and LTX-2, skipping redundant computation across diffusion steps
• Prompt embedding caching means repeated prompts no longer re-encode text on every generation request

New models added to the image/video stack include HiDream-I1-Full, SenseNova U1, Ming-flash-omni-2.0 image generation, and Lance (ByteDance).

Quantization and hardware coverage

This is where the release gets particularly useful for teams running on non-standard hardware. The quantization matrix has expanded substantially:

For NVIDIA Blackwell GPUs (GB200, B200, RTX 5090, DGX Spark), the release adds CUDNN attention, FlashInfer attention auto-routing, and SageAttention3 diffusion backend support. AMD ROCm gets AITER GroupNorm and AITER ring attention backend support. Intel XPU picks up CosyVoice3 and MXFP8. Ascend NPU gets Yuanrong TransferEngine connector support for high-speed inter-node data movement.

Architecture: the OmniCoordinator

Under the hood, the most significant architectural change is the integration of OmniCoordinator into the stage engine pipeline. vLLM-Omni's core abstraction is the stage , a separate model runner for each modality in a pipeline (e.g., a text encoder stage, a diffusion stage, an audio decoder stage). The OmniCoordinator handles routing requests across these stages, managing memory handoffs, and scaling replicas independently. This release makes that coordination layer more robust: better request routing, model-runner reuse, and a communication-layer refactor that reduces overhead in non-async paths.

The release also aligns with the vLLM 0.22 release line, inheriting CUDA 13.0 support, PyTorch 2.11, and HuggingFace Transformers 5.x compatibility.

Getting started

vLLM-Omni is free and open-source under the Apache 2.0 license. You can run it via Docker with a command like docker run --runtime nvidia --gpus 2 vllm/vllm-omni:v0.22.0 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091. Since vllm-omni is rapidly evolving, it is recommended to install it from source with git clone https://github.com/vllm-project/vllm-omni.git && cd vllm-omni && uv pip install -e .

The practical use cases this release unlocks are concrete: production TTS APIs with voice cloning, robot policy serving for real-time closed-loop control, synthetic training data generation for physical AI using Cosmos 3, and high-throughput video generation with quantized diffusion models. vLLM-Omni is one of the first open-source frameworks to support omnimodal model serving that extends vLLM's exceptional performance to the world of multi-modal and non-autoregressive inference. With world models and robot serving now in the stack, the gap between research demos and production deployment just got a lot narrower.