NVIDIA's Cosmos 3 Merges Four Robot AI Systems Into One Open Model
NVIDIA's Cosmos 3 merges reasoning, world simulation, and robot action generation into a single open model — a first for physical AI
Takeaways
- Cosmos 3 is the first fully open omnimodal model for physical AI, unifying vision reasoning, world simulation, and robot action generation in a single architecture.
- Two sizes available now: Nano (8B/16B MoT) for edge/workstation inference and Super (32B/64B MoT) for datacenter-scale synthetic data generation.
- New MoT architecture pairs an autoregressive reasoner tower with a diffusion-based generator tower, replacing the need to orchestrate multiple separate models.
- #1 open-source model on Artificial Analysis Text-to-Image and Image-to-Video leaderboards, PAI-Bench, R-Bench, Physics-IQ, RoboLab, and RoboArena.
- Six synthetic datasets released covering robotics, autonomous driving, warehouse operations, spatial reasoning, physics simulation, and digital humans — all on Hugging Face.
- Known failure modes include temporal inconsistency, object morphing, and implausible physics — safety-critical applications require additional validation before deployment.
Building a robot that can reliably pick up an object sounds deceptively simple. In practice, it requires stitching together at least four separate systems: a vision model to parse the scene, a reasoning model to plan the move, a dynamics model to predict what happens next, and a policy model to output motor commands. Each component is trained separately, stitched together with glue code, and prone to compounding errors at every handoff. NVIDIA's answer to this fragmentation is Cosmos 3, a single open model that does all of it at once.
One model to rule the physical world
The biggest change compared to previous Cosmos releases is that it's an omni-model, built on a Mixture-of-Transformers (MoT) architecture. Previously, developers had to work with separate models for world generation (Cosmos Predict), controlled generation (Cosmos Transfer), scene understanding (Cosmos Reason), and policy generation (Cosmos Policy). Cosmos 3 enables all of this in a single model that can reason and generate different modalities in one unified forward pass.
Cosmos 3 adopts a Mixture-of-Transformers (MoT) architecture that processes a unified sequence of tokens from different modalities. MoT is distinct from the more familiar Mixture-of-Experts (MoE) approach: rather than routing tokens to different expert sub-networks, MoT maintains two full sets of parameters at every transformer layer, one for reasoning and one for generation, and activates both during generation tasks.
The two-tower trick
The architecture is organized around two towers that work in tandem:
- Reasoner tower: An autoregressive vision-language model (VLM) that interprets text, images, and video. It acts as the brain, understanding motion, object interactions, and physical context before any generation happens.
- Generator tower: A diffusion-based model that produces physics-aware video and action outputs, always conditioned on what the reasoner tower has understood.
In Reasoner Mode, language and visual understanding tokens are processed through causal self-attention, enabling next-token prediction for tasks such as perception, planning, and world reasoning. In Generator Mode, noisy image, video, audio, and action tokens are denoised through full attention, allowing the model to jointly generate coherent multimodal outputs.
Training Cosmos 3 requires data for two complementary objectives: the Reasoner pathway learns to understand and reason about the world, while the Generator pathway learns to synthesize and simulate it, or act within it. Nano and Super are initialized from pre-trained Qwen3-VL weights. The action training is especially nuanced: forward dynamics mode denoises vision tokens conditioned on clean action tokens; inverse dynamics mode denoises action tokens conditioned on clean vision tokens; and video-action (policy) mode denoises both vision and action tokens.
What's in the box
Two model sizes ship today, with a third on the way:
- Cosmos 3 Nano (8B dense / 16B MoT): Designed for workstation-grade hardware like the NVIDIA RTX PRO 6000. Targets real-time robotics inference at the edge.
- Cosmos 3 Super (32B dense / 64B MoT): Built for datacenter deployment on Hopper and Blackwell GPUs. Optimized for large-scale synthetic data generation and advanced physical reasoning.
- Cosmos 3 Edge (2B dense / 4B MoT): Will be included in a later release. Targets on-device deployment.
The full release includes model weights, post-training recipes, training code, and six open synthetic datasets on Hugging Face. Everything is available under the Linux Foundation's OpenMDW-1.1 License. Note that this is not Apache 2.0 , check the license terms before production use.
Benchmarks: leading the open-source pack
Cosmos 3 models were ranked as the best open-source Text-to-Image and Image-to-Video models by Artificial Analysis, and the best policy model by RoboArena at the time the technical report was written. On the physical AI side:
- PAI-Bench, R-Bench, Physics-IQ, RoboLab: Cosmos 3 is the open-source SOTA and currently leads across these public leaderboards.
- VANTAGE-Bench: Cosmos 3 Super and Cosmos 3 Nano lead at the 32B and 8B tiers respectively on this benchmark for vision-language models evaluated on real-world fixed-camera footage across warehouses, transportation, and smart spaces.
- TAR (Traffic Anomaly Reasoning): Leads on the official AI City Challenge 2026 Track 3 leaderboard for detecting anomalous events in transportation footage.
A word of caution: benchmark performance and production reliability in messy real-world environments are different things. Independent evaluations are still pending for some of these claims.
Where it shines, where it struggles
Cosmos 3 is purpose-built for physical AI workflows. The practical sweet spots are:
- Generating large volumes of synthetic training data for robots and autonomous vehicles
- Simulating rare or dangerous edge cases (warehouse accidents, traffic anomalies) that are hard to capture in the real world
- Learning robot manipulation policies from video demonstrations without manually labeling actions
- Sub-second visual reasoning for scene understanding in smart infrastructure
The model has documented limitations. Common failure modes include temporal inconsistency, unstable camera or object motion, inaccurate sound-video alignment, imperfect action-state consistency, object morphing, inaccurate 3D structure, and implausible physical dynamics. Applications that require physically grounded simulation, safety-critical control, or complex multi-agent behavior need additional validation, guardrails, and system-level safety analysis before deployment.
Six datasets, one ecosystem
Alongside the model, NVIDIA is open-sourcing six synthetic data generation datasets on Hugging Face. These are immediately useful for post-training Cosmos 3 on specific domains:
- Embodied Robot Scenes , humanoid robots doing manipulation tasks
- Physical Interaction Scenes , simulated physics with ground-truth annotations (velocity, center-of-mass, segmentation)
- Spatial Reasoning , indoor environments with Q&A pairs for navigation and distance estimation
- Digital Human Scenes , diverse human motion, appearance, and lighting
- Autonomous Driving Scenarios , ego-view driving across weather conditions and behaviors
- Warehouse Operations Scenes , forklift interactions, safety incidents, multi-camera views
Getting it running
There are two main paths to deployment. For inference workflows, NVIDIA offers NIM microservices , pre-packaged containers with optimized runtimes. The Cosmos 3 Reasoner NIM is available today. Spinning up the Nano reasoner is a single Docker command:
docker run --gpus=all \
-e NGC_API_KEY=$NGC_API_KEY \
-e NIM_MODEL_SIZE=nano \
-p 8000:8000 \
nvcr.io/nim/nvidia/cosmos3-reasoner:latestThe NIM supports BF16, FP8, or NVFP4 quantized checkpoints, with NVFP4 achieving up to 2x inference speedup. For post-training on your own data, the Cosmos GitHub repo is the right starting point, with configs for supervised fine-tuning on custom video datasets and action-labeled robotics data.
The Hugging Face collection also includes a Cosmos3OmniPipeline class via Diffusers, which makes it straightforward to integrate into existing generation workflows.
The bigger picture
NVIDIA also launched the Cosmos Coalition alongside the model , a group of partners including Agile Robots, Black Forest Labs, Runway, and Skild AI , focused on sharing evaluation techniques, training data, and research around open world model development. This is part of a broader strategic move: NVIDIA is building a vertically complete open physical AI stack, from the edge device to the world model, and releasing the software layer open-source to accelerate adoption.
The field has long assumed that world models, action models, and reasoning systems need to be separate architectures trained on separate data pipelines. Cosmos 3 is a direct challenge to that assumption. For physical AI workloads in particular, where proprietary sensor data, safety validation, and operational continuity are all in play, the open model framing may appeal less to established industrial manufacturers than to the startup and research community , at least for now. But for teams building the next generation of robot brains, this is the most capable open foundation available.