Skip to main content
Back

NVIDIA's Cosmos 3 Merges Robot Reasoning and Video Into One Open Model

NVIDIA's Cosmos 3 is the first open omni-model that unifies physical reasoning, world generation, and robot action prediction in a single architecture — and it just topped the open-weights leaderboards.

  • NVIDIA Cosmos 3 is the first fully open omni-model for physical AI, unifying reasoning, video generation, and robot action prediction in one model.
  • Built on a Mixture-of-Transformers (MoT) architecture pairing an autoregressive reasoner with a diffusion generator; comes in 16B (Nano) and 64B (Super) sizes.
  • Tops open-weights leaderboards on Artificial Analysis for both Text-to-Image and Image-to-Video, beating FLUX.1 [dev], Wan 2.2, and LTX-2.
  • Fully open under OpenMDW 1.1 with weights, training code, fine-tuning recipes, and 6 synthetic data generation datasets on Hugging Face.
  • Supports text, image, video, audio, and action as both inputs and outputs; generator requires structured JSON prompts with prompt upsampling.
  • Known limitations include temporal inconsistency, object morphing, and sound-video misalignment; safety-critical deployments need additional validation.

NVIDIA just shipped something the robotics and physical AI world has been waiting for: a single open model that can reason about the physical world, generate physics-accurate video, and output robot action sequences , all without switching between separate pipelines. Cosmos 3 launched at GTC Taipei and immediately claimed the top open-weights spot on both the Artificial Analysis Text-to-Image and Image-to-Video leaderboards, beating out HiDream, Alibaba's Qwen Image Max, Black Forest Labs' FLUX.1 [dev], and Lightricks' LTX-2.

The problem it's solving

Consider a home robot instructed to clean a dining table after dinner. Under the current paradigm, the robot must stitch together a disjointed suite of models: a VLM to locate dishware and generate an executable plan, a VLA or WAM to generate action sequences, and a forward dynamics model to simulate and evaluate future states. This fragmented architecture is suboptimal and computationally wasteful.

The biggest change in Cosmos 3 compared to previous Cosmos releases is that it's an omni-model. Previously, developers had to work with separate models for different capabilities like world generation (Cosmos Predict), controlled generation (Cosmos Transfer), scene understanding (Cosmos Reason), and policy generation (Cosmos Policy). Cosmos 3 enables all of this in a single model that can reason and generate different modalities in one unified forward pass.

One architecture to rule them all

Cosmos 3 is an omnimodal world model built on a unified Mixture-of-Transformers (MoT) architecture that combines an autoregressive (AR) transformer for reasoning with a diffusion transformer (DM) for multimodal generation. The MoT design , think of it as two specialist sub-networks sharing the same backbone , is what makes this possible without a massive compute penalty.

Here's how the two towers divide the work:

  • Reasoner tower: A vision-language model that interprets multimodal observations like images, videos, and text. It uses an autoregressive architecture to understand motion, object interactions, and physical context , serving as the model's brain before any generation happens.
  • Generator tower: Generates future observations and action sequences using a diffusion-based process to produce physics-aware video and action outputs conditioned on the reasoner tower's understanding. The reasoner can be called independently, but the generator always activates both towers for guided generation.

AR and DM tokens use separate parameter sets within each transformer layer but interact through joint attention , this is what lets a single model seamlessly switch between acting as a VLM, a video generator, a forward/inverse dynamics model, or a robot policy without any architectural changes.

The architecture also introduces a 3D multimodal rotary position embedding (mRoPE) , a way of encoding where things are in space and time , that aligns video, audio, and action tokens on one shared temporal axis. During inference, text is generated through standard next-token autoregressive decoding, while non-text modalities such as images, video, audio, and actions are synthesized through iterative denoising.

Model sizes and what runs where

The family ships in two sizes today, with a third on the way:

  • Cosmos 3 Nano (16B): The compact version with 16B parameters, optimized for efficient inference. It's designed to run on workstation-grade compute like the NVIDIA RTX PRO 6000 GPU for real-time robotics inference.
  • Cosmos 3 Super (64B): A 64B parameter model designed for maximum quality and capability, targeting datacenter Hopper and Blackwell GPUs for large-scale synthetic data generation.
  • Cosmos 3 Edge (4B): An edge model that can run locally is coming soon.

Both Nano and Super are initialized from pre-trained Qwen3-VL weights, which is why the total parameter count roughly doubles the backbone , each tower is a full copy of the base transformer. Task-specific fine-tunes also ship: Cosmos3-Super-Text2Image and Cosmos3-Super-Image2Video are the variants that topped the Artificial Analysis leaderboards.

What it can actually do

By supporting highly flexible input-output configurations, Cosmos 3 seamlessly unifies critical modalities for Physical AI , effectively subsuming vision-language models, video generators, world simulators, and world-action models into a single framework. This covers world understanding, grounding, physical reasoning, task planning, action forecasting, embodied agent reasoning, and autonomous system decision making.

A quick look at the supported input/output combinations:

InputOutputUse Case
Text / Image / VideoVideoWorld simulation
Text / VideoTextVision-language reasoning (VLM)
Action + Image + TextVideoForward dynamics (what happens if robot does X)
Text / VideoActionInverse dynamics (what action caused this?)
Image + TextVideo + ActionRobot policy generation

One important quirk: the generator takes structured JSON prompts rather than plain text. Short natural-language descriptions need to be expanded into dense structured prompts first , a process called prompt upsampling. The generator can be driven either by an external prompt-upsampling harness or by its own reasoner branch, so it can run self-contained.

Benchmark results

On reasoning, Super and Nano lead VANTAGE-Bench at their respective tiers. VANTAGE-Bench tests VLMs on real-world fixed-camera footage, covering warehouses, transportation, and smart spaces. Cosmos 3 also tops the Traffic Anomaly Reasoning (TAR) leaderboard.

On generation, NVIDIA reports open-source state-of-the-art results. Cosmos 3 is the open-source SOTA on R-Bench and also leads PAI-Bench, Physics-IQ, and RoboLab on public leaderboards. On the Artificial Analysis arena, Cosmos3-Super-Text2Image ranks #4 overall in text-to-image (across all models, open and closed), and #1 among open-weights models in both text-to-image and image-to-video.

Where it falls short

The model card and technical documentation are candid about current limitations. Generated outputs can exhibit:

  • Temporal inconsistency and unstable motion across frames
  • Object morphing mid-sequence
  • Inaccurate 3D structure
  • Sound-video misalignment in audio generation

Cosmos 3 understands and generates world videos and actions for robotics, autonomous vehicles, and smart spaces out of the box, but some applications may require further post-training on specific datasets to get the best results. Safety-critical deployments , anything where a robot is making real-world decisions , will need additional validation, guardrails, and system-level analysis before production use.

Getting started

The model is fully open under the OpenMDW 1.1 license. Cosmos 3 is available now on Hugging Face with weights, code, curated datasets, and fine-tuning recipes. The quickest path to running it is through the Diffusers integration:

import torch
from diffusers import Cosmos3OmniPipeline
pipe = Cosmos3OmniPipeline.from_pretrained(
    "nvidia/Cosmos3-Nano", torch_dtype=torch.bfloat16, device_map="cuda"
)
result = pipe(
    prompt="A robotic arm on a white workbench, gripper above colored objects...",
    num_frames=1,
    height=720,
    width=1280
)
result.video[0].save("cosmos3_t2i.jpg", format="JPEG", quality=85)

Cosmos 3 models are also available as NVIDIA NIM microservices for optimized, production-ready deployment. NIM microservices package the model with optimized inference runtimes, delivering high performance without the need to manually tune serving infrastructure. The Reasoner NIM is available now; the Generator NIM follows shortly. First-party and third-party APIs with pricing are expected over the coming weeks.

Six synthetic data generation (SDG) datasets ship alongside the model, covering robotics simulation, physics interaction, spatial reasoning, human motion, autonomous driving, and warehouse safety scenarios , all on Hugging Face. The GitHub repo also includes post-training scripts for adapting the model to your own robot embodiments and environments.

The bigger picture

World models have become a key growth area for AI as companies increasingly want to take the smarts of chatbots and agents and allow them to perform real-world tasks. NVIDIA Cosmos 3, launched as the first open omni-model for physical AI reasoning and action, aims to accelerate robotics and embodied AI by providing a unified, open-source foundation. This move challenges proprietary approaches from OpenAI, Google, and Amazon, but faces hurdles in standardization, integration, and ecosystem buy-in.

The release also comes with the NVIDIA Cosmos Coalition , a global collaboration between world model builders and AI developers including Agile Robots, Black Forest Labs, Generalist, LTX, Runway, and Skild AI , working together to advance next-generation world models. The model can natively understand and generate text, images, video, ambient sound and actions with leading physics accuracy, reducing physical AI training and evaluation cycles from months to days. That last claim is the one worth watching: if it holds up at scale, the bottleneck for training physical AI systems shifts from data collection to compute , and that's a very different problem to have.