NVIDIA's Nemotron 3 Ultra Runs on Half the Hardware at 5x Faster Speed
NVIDIA's Nemotron 3 Ultra is a 550B open-weight MoE model that delivers 5x faster inference and 30% lower cost for agentic workloads, now the top US open-weight model by benchmark.
Takeaways
- Released: Nemotron 3 Ultra is a 550B-parameter open MoE model (55B active) built for long-running agentic workflows, now available on Hugging Face and build.nvidia.com.
- Speed & cost: Claims 5x faster inference and 30% lower cost per agentic task vs. comparable open models, serving 400+ tokens/sec on BlackBox AI endpoints.
- Top US open model: Scores 48 on the Artificial Analysis Intelligence Index, ahead of Gemma 4 31B (39) and GPT-OSS 120B (33), but trails China's Kimi K2.6 (54).
- Key innovations: Hybrid Mamba-Transformer architecture, LatentMoE routing, multi-token prediction, NVFP4 quantization, and a novel Multi-Teacher On-Policy Distillation (MOPD) training method.
- Fully open: Weights, training data, RL environments, and recipes released under the permissive OpenMDW-1.1 (Linux Foundation) license; fine-tunable via LoRA, SFT, or RL.
- Weaknesses: Trails Kimi K2.6 and GLM 5.1 on raw coding (Terminal-Bench 2.0: 54% vs 67%) and long-horizon planning (EnterpriseOps-Gym: 33% vs 40%).
NVIDIA just shipped its biggest open model yet, and the pitch is not about topping a leaderboard. Nemotron 3 Ultra is a 550B-parameter Mixture-of-Experts model with 55B active parameters, built for frontier reasoning and orchestration in agentic systems. The design goal is explicit: not winning single-turn chatbot comparisons, but orchestrating agents that plan, call tools, delegate to sub-agents, read observations, and recover from errors across hundreds of turns, while burning fewer tokens than comparable open models.
Nemotron 3 Ultra was announced at Computex 2026 and released on June 4. NVIDIA released the base model weights, post-trained checkpoints, reward models, NVFP4 quantized variants, training recipes, and datasets under the OpenMDW-1.1 license, a permissive open AI model license from the Linux Foundation. That makes it one of the most complete open releases from a major AI lab in recent memory.
The Biggest Open US Model, But Not the Biggest Overall
Nemotron 3 Ultra scores 47.7 on the Artificial Analysis Intelligence Index, well ahead of the next strongest US open weights models: Gemma 4 31B (39.2), Nemotron 3 Super (36.0), and gpt-oss-120b (33.3), but behind the Chinese-led open weights frontier, Kimi K2.6 at 53.9. The gap with China is real, but the efficiency story is where NVIDIA makes its case.
Through BlackBox AI ahead of release, Nemotron 3 Ultra is served at over 400 output tokens per second. That is three to six times faster than comparably sized open models from other labs. NVIDIA claims 5x faster inference and a 30% cost reduction for agentic tasks. The combination of low active parameter count, NVFP4 quantization, and architectural innovations is what makes those numbers possible.
Five Architectural Bets Working Together
Architectural innovations include hybrid Mamba-Transformer layers for efficient long-context handling, NVFP4 quantization for cross-architecture GPU deployment with up to 5x higher throughput, LatentMoE for expert routing, and multi-token prediction for improved generative speed in multi-turn tasks. Each of these is worth unpacking:
- Hybrid Mamba-Transformer: Mamba layers (a type of state-space model that processes sequences more efficiently than attention) handle long-context workloads cheaply, while Transformer attention layers are kept where precise recall matters. This is what makes the 1M token context window practical without exploding cost.
- LatentMoE: Instead of routing tokens to experts in the full hidden dimension, Ultra first projects tokens into a smaller latent space, then routes. This reduces routing overhead and lets a single model handle reasoning, code generation, tool calls, and domain-specific logic efficiently.
- Multi-token prediction (MTP): The model predicts several future tokens in a single forward pass using shared weight prediction heads, giving it native speculative decoding and cutting latency on long outputs and multi-turn workflows.
- NVFP4 precision: The same checkpoint runs across NVIDIA Hopper, Blackwell, and Ampere GPUs. Nemotron 3 Ultra was pre-trained in NVFP4, so the low-precision checkpoint holds accuracy without a separate calibration step.
The model supports context lengths of up to 1M tokens while outperforming state-of-the-art open LLMs on RULER at 1M context length. That 95% RULER score at 1M tokens is a structural advantage: both GLM 5.1 and Kimi K2.6 cap out at 256K and cannot run the test at all.
A New Training Method: Multi-Teacher On-Policy Distillation
The most novel piece of the training pipeline is what NVIDIA calls Multi-Teacher On-Policy Distillation (MOPD). The idea: instead of learning from a single teacher, Ultra learns from dense feedback from over ten domain-specific teacher models. Each teacher is trained with its own domain-specific pipeline, covering coding, math, search, office work, and tool calling. As the student model generates rollouts, the matching teacher scores it in its area of expertise and feeds back a reward signal.
The process runs asynchronously, with student rollout generation, teacher scoring, and student optimization fully pipelined. It is also iterative: after producing an MOPD-trained checkpoint, new rounds of teacher training are initialized from the updated student, then merged into the next MOPD stage. Student and teachers co-evolve, which lets capability keep climbing rather than plateauing. The teacher lineup includes Qwen3 variants, DeepSeek V3, R1, V4-Pro, GPT-OSS-120B, GLM-5, GLM-4.7, Mistral variants, and phi-4.
The earlier stages are more conventional. Pre-training runs on roughly 20 trillion tokens with a September 2025 cutoff. The model underwent multi-phase post-training: SFT for Math, Code, Reasoning, Chat, and Tool Calling, then RL with GRPO to enhance reasoning and instruction-following. MOPD sits on top as the final refinement layer, and NVIDIA has released the MOPD recipe through NeMo-RL.
NVIDIA is releasing the pre-trained, post-trained, and quantized checkpoints along with the datasets used for training. The NVFP4 post-trained model is the recommended inference checkpoint. Building on its existing pre-training foundation, Nemotron 3 Ultra also added 212B new tokens targeting three specific domain gaps:
- 4B tokens of synthetic legal data, lifting the proxy LegalBench average from 64.6% to 74.7%
- 35B tokens of synthesized Wiki-based data, boosting proxy SimpleQA from 40.2% to 50.2%
- 173B refreshed GitHub tokens through September 30, 2025
Where It Wins and Where It Doesn't
The benchmark picture is honest about the tradeoffs. Here is how Ultra stacks up against the strongest open models on NVIDIA's own evaluation suite:
| Benchmark | Nemotron 3 Ultra (550B) | GLM 5.1 (744B) | Kimi K2.6 (1T) | Qwen3.5 (397B) |
|---|---|---|---|---|
| Agent Productivity (PinchBench) | 91% | 84% | 91% | 89% |
| Long-horizon Planning (EnterpriseOps-Gym) | 33% | 40% | 29% | 30% |
| Coding (Terminal-Bench 2.0) | 54% | 64% | 67% | 53% |
| Instruction Following (IFBench) | 82% | 77% | 74% | 78% |
| Long Context (RULER @1M) | 95% | N/A (max 256K) | N/A (max 256K) | 90% |
Performance on CritPt, a benchmark for graduate-level physics research problems, is relatively lower, reaching only 3%, the same as Nemotron 3 Super. Nemotron 3 Ultra scored 71% on AA-Omniscience Non-Hallucination, reflecting a low propensity to answer factual questions when it doesn't know the answer. The honest read: Ultra wins on agent productivity, instruction following, long context, speed, and cost, and trails on raw coding, deep planning, and peak knowledge work.
The Agent Stack It Ships With
Nemotron 3 Ultra does not drop as a bare model. NVIDIA released it alongside a reference stack for running autonomous agents more safely. Hermes Agent and OpenClaw are the supported orchestration harnesses. NVIDIA OpenShell, in early preview as part of the NVIDIA Agent Toolkit, is the secure runtime where autonomous agents and their generated code actually execute. NVIDIA NemoClaw is the open-source blueprint that ties it together, installing the OpenShell runtime with a single command.
Two companion models also launched alongside Ultra:
- Nemotron 3.5 Content Safety: An open 4B guardrail model covering 23 safety categories and 12 languages, usable as an inference-time filter, an LLM safety judge, or a fine-tuning base for custom safety policies.
- Nemotron 3.5 ASR: A multilingual speech recognition model with sub-100ms streaming latency, extending the English architecture that already powers voice input in Microsoft GitHub Copilot CLI to 40+ languages in a single checkpoint.
The Real Shift: Efficiency as a Competitive Axis
The broader context matters here. For most of the past year, the strongest open models came out of Chinese labs. Nemotron 3 Ultra scores 48 on the Artificial Analysis Intelligence Index v4.0, the highest score of any US-developed open-weight model as of this release, ahead of Gemma 4 31B (39), Nemotron 3 Super (36), and GPT-OSS 120B (33). The gap with China's open frontier narrowed, but the more important signal is the competitive axis NVIDIA chose.
Rather than racing to the top of a single benchmark, NVIDIA built a model optimized for cost per completed task. DeepSeek R1 in BF16 needs approximately 1.34 TB of VRAM for weights alone, requiring 16-18 H100s minimum. Nemotron Ultra at FP8 fits on 8 H100s. For a team running their own inference infrastructure, that's half the hardware and half the monthly cost for comparable or better benchmark scores on most tasks. As agent workflows grow longer and token bills compound, that framing is increasingly the one enterprises actually optimize for.
How to Get It Running
NVIDIA is releasing the pre-trained, post-trained, and quantized checkpoints on Hugging Face. You can also try it immediately via build.nvidia.com, OpenRouter, Perplexity Pro, or Anaconda. For self-hosting, the model ships with deployment recipes for vLLM, SGLang, and TensorRT-LLM. A quick vLLM launch looks like this:
docker run --gpus all -p 8000:8000 \
vllm/vllm-openai:latest \
--model nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 \
--tensor-parallel-size 8 \
--quantization fp8 \
--max-model-len 32768 \
--enable-chunked-prefill \
--gpu-memory-utilization 0.92Reasoning mode is configurable at inference time through the chat template: full thinking, medium effort, or reasoning off. You can also set a hard token budget on the thinking trace, which is a practical control for agent builders watching token spend. Fine-tuning is supported via LoRA, full SFT, and RL through the NeMo libraries, with H100 and GB200 recipes available in the GitHub repository. Pricing is set by each inference provider since the model is open, but the throughput advantage means effective cost per task lands well below comparable open models even before the 30% token reduction NVIDIA claims.