NVIDIA's Cosmos3-Nano Merges Video, Audio, and Robot Control in One 16B Model
NVIDIA's 16B omnimodal world model unifies text, image, video, audio, and robot action generation under one Mixture-of-Transformers architecture.
- Cosmos3-Nano is a 16B omnimodal world model for Physical AI applications.
- Handles text, image, video, audio, and robot action as both input and output.
- Built on a Mixture-of-Transformers combining autoregressive text and diffusion for other modalities.
- Trained on 1.3B samples including 348M videos and robot data from AgiBot, Franka, UR.
- Released under permissive OpenMDW 1.1 license; supported in vLLM-Omni, Diffusers, SGLang.
- Supports forward and inverse dynamics on 10 robot embodiments including autonomous vehicles.
NVIDIA has released the Cosmos3-Nano model card, a 16 billion parameter world model that accepts and generates text, images, video, audio, and action trajectories. NVIDIA positions it for Physical AI, its term for systems that perceive and act in environments such as robots, autonomous vehicles, and instrumented spaces.
Action trajectories give the release practical significance for developers. The model can predict future observations from proposed controls, estimate controls from observed video, and generate multimodal training data within one framework. Cosmos3-Nano is the smallest member of the Cosmos 3 family. Its OpenMDW 1.1 license permits commercial and non-commercial use, subject to the license terms.
Two engines split the work
Cosmos3-Nano uses a Mixture-of-Transformers architecture with two complementary generation systems. An autoregressive transformer produces discrete text tokens one at a time. A diffusion transformer generates continuous outputs such as images, video, audio, and actions by repeatedly refining noise into a coherent signal. The components share multimodal representations and conditioning within the same framework.
| Path | Primary role | Reported limits |
|---|---|---|
| Generator | Text, image, video, audio, and action generation | 4,096 text tokens; image and video up to 720p; stereo audio at 48 kHz |
| Reasoner | Structured reasoning, planning, point localization, and bounding boxes | 256K-token context; video sampling recommended at 4 fps |
Action data uses a two-dimensional array shaped (T, D), where T is the number of frames and D is the state dimension for a specific robot or vehicle. Each frame can encode values such as joint positions, gripper state, camera pose, or steering and acceleration commands.
The supplied action schemas cover several embodiments:
- General camera motion and autonomous vehicle control, both 9D
- Egocentric human motion, 57D
- Single and dual Franka Panda arms with RobotiQ grippers, 10D and 20D
- AgiBot humanoid, 29D
- Universal Robots, Google robot, and WidowX 250, each 10D
- UMI robot demonstrations, 9D
The Reasoner path emits structured rationales, subtask plans, 2D or 3D points, and bounding boxes. These outputs can supply a downstream perception or planning system, although generated reasoning traces remain model outputs rather than evidence of reliable internal reasoning.
Benchmarks span reasoning to control
NVIDIA reports comparisons with Gemini, Qwen, and open video-generation models across general reasoning, robotics, smart infrastructure, autonomous driving, and multimodal generation. The published charts place Cosmos3-Nano in the same evaluation set as larger general-purpose systems, although the results remain vendor-reported until independently reproduced.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
