NVIDIA's Nemotron 3 Ultra Tops US Open-Weight AI on Brutal New Job Benchmark
Artificial Analysis launches AA-Briefcase, a new agentic benchmark for long-horizon knowledge work, where NVIDIA's Nemotron 3 Ultra ranks among the top open models
Takeaways
- Artificial Analysis launched AA-Briefcase, a new agentic benchmark testing models on 91 tasks across 4 multi-week knowledge work projects.
- NVIDIA's Nemotron 3 Ultra (550B total / 55B active MoE) ranks among the top open-weight models on the new leaderboard.
- Nemotron 3 Ultra scores 48 on the Artificial Analysis Intelligence Index -- the highest of any US-developed open-weight model, but 6 points behind China's Kimi K2.6 (54).
- The model runs 3-6x faster than comparable open-weight rivals and lowers agentic task costs by up to 30% through token efficiency.
- Even the top-ranked model on AA-Briefcase (Claude Fable 5) passes all rubric checks on only 3% of tasks -- highlighting how hard real knowledge work remains for current agents.
- Nemotron 3 Ultra is available now on Hugging Face, OpenRouter, NVIDIA NIM, and 25+ platforms under the permissive OpenMDW-1.1 license.
Most AI benchmarks test a model's ability to answer a question. AA-Briefcase tests whether a model can hold down a job. Artificial Analysis just launched the benchmark, and it is one of the most demanding evaluations of real-world agentic capability published to date. NVIDIA was quick to highlight that its Nemotron 3 Ultra ranks among the top open-weight models on the new leaderboard.
What AA-Briefcase Actually Tests
The benchmark is not a collection of isolated prompts. AA-Briefcase evaluates models across four multi-week knowledge work projects, comprising thousands of input files and 91 tasks in total. Across the scenarios, models must complete realistic professional workflows in fields such as data science, product management, and corporate strategy.
The four held-out scenarios cover:
- Data Science -- transaction data cleaning, forecast modeling, and schema design
- Product Management -- competitive teardowns, PRD writing, and go-to-market planning
- Banking Operations -- branch traffic analysis, financial modeling, and mortgage journey mapping
- Heavy Industry Strategy -- commodity price modeling, M&A comps, and government policy impact matrices
All four launch scenarios remain held out to reduce contamination risk and preserve AA-Briefcase as a reliable measure of frontier agent capability. A fifth public scenario called AA-Briefcase Lite is available on Hugging Face for anyone who wants to inspect the structure without influencing official scores.
What makes this benchmark unusually hard is the volume and messiness of the source material. AA-Briefcase is a private evaluation developed by Artificial Analysis for frontier agentic capability in long-horizon knowledge work, testing agents on realistic business workflows that require deliverables such as spreadsheets, presentations, and memos. The source material includes nearly 2,000 files, over 3,500 emails, and 25,000 Slack messages -- fragmented, contradictory, and realistic.
How the Scoring Works
AA-Briefcase Elo is a combined metric that aggregates rubric pass rate, analytical quality Elo, and presentation Elo. Rubric performance is converted into Elo via synthetic head-to-head matches. Think of it as a chess-style rating that combines three things: did the model get the facts right (rubric), was the analysis rigorous (analytical quality), and did the output look professional (presentation).
Grading is handled by a rotating panel of three frontier LLM judges -- Claude Opus 4.8, GPT-5.5, and Gemini 3.1 Pro Preview -- rather than a single model, reducing bias toward submissions from the same model or model family.
The results reveal something humbling about the current state of agentic AI. Even the top-ranked model, Claude Fable 5, satisfies all rubric criteria correctly on only 3% of tasks. On 31 of the 91 tasks, no model scores above 50%. This is not a benchmark where anyone is acing the exam.
Where Nemotron 3 Ultra Fits In
NVIDIA released Nemotron 3 Ultra, a 550B-parameter Mixture-of-Experts model with 55B active parameters, optimized for orchestrating complex, long-running agent workflows by combining frontier reasoning and high throughput with domain adaptability. It was announced at Computex 2026 and released on Hugging Face shortly after.
The architecture is worth understanding because it directly explains the model's performance profile:
- Hybrid Mamba-Transformer -- Mamba layers handle long-context sequences efficiently; Transformer layers handle dense reasoning. NVIDIA's bet is that Mamba layers handle efficient long-range sequence modeling (critical for 200k+ token contexts), while transformer layers handle the dense reasoning and attention patterns that benchmarks reward.
- Mixture-of-Experts (MoE) -- Only 55B of the 550B parameters activate per token, a 10x sparsity ratio that keeps inference fast without sacrificing capacity.
- NVFP4 quantization -- A 4-bit floating point format that delivers up to 5x higher throughput for cross-architecture GPU deployment.
- Multi-Token Prediction (MTP) -- Predicts multiple future tokens in a single forward pass, improving throughput in multi-turn agentic loops.
The training story is equally distinctive. Multi-Teacher On-Policy Distillation enables continuous improvement and domain specialization by training Nemotron 3 Ultra with dense feedback from over ten domain-specific teacher models, supported by a massive and transparent pretraining and RL data pipeline, with fully open recipes, weights, and licensing.
The Numbers That Matter
On the broader Artificial Analysis Intelligence Index, Nemotron 3 Ultra scores 48 -- the highest score of any US-developed open-weight model as of June 2026. That is a significant jump: Nemotron 3 Ultra doesn't just edge past competing US open-weight models -- it resets the American open-weight ceiling by 9 full points.
Speed is where Nemotron 3 Ultra really separates itself. Chinese open-weight models of similar intelligence -- DeepSeek V4 Pro and Kimi K2.6 -- run at 50-100 tokens per second through their commercial APIs, making Nemotron 3 Ultra three to six times faster in practice. For agents running hundreds of turns per task, that gap compounds quickly.
On cost efficiency, in experiments on SWE-bench and Terminal Bench 2.0, it completed benchmarks using fewer total tokens and fewer tokens per turn than comparable models, lowering the cost for agentic tasks by up to 30%.
The honest benchmark picture is more nuanced though. Nemotron 3 Ultra places six points below China's Kimi K2.6, which scores 54 on the Intelligence Index. On AA-Briefcase specifically, GLM-5.2 (max) is the clear leader among open-weight models and offers an attractive agentic capability vs. cost tradeoff. Nemotron 3 Ultra ranks among the top open models, but the leaderboard is competitive.
What the Benchmark Reveals About Agents in General
AA-Briefcase surfaces some patterns that go beyond any single model's score. Task difficulty scales directly with how many source files a model must read -- high-intelligence models fall from roughly 55% pass rate on prompt-only checks to around 40% when a task requires pulling from 5 or more external files. Even the best models struggle to synthesize fragmented context at scale.
There is also a striking finding about presentation quality. The two leading models on presentation Elo -- Claude Fable 5 and Claude Opus 4.8 -- make 21 and 12 visual inspections of their rendered outputs per task respectively. Models that rarely inspect their own outputs before submitting score significantly lower on presentation. The implication: agents that "check their work" visually produce better deliverables.
AA-Briefcase runs on Stirrup, Artificial Analysis's open-source agent framework. The leaderboard is live and updated as new models are submitted. For teams building or evaluating agents for real knowledge work -- not just coding or Q&A -- this is now one of the most relevant benchmarks to watch.
Availability and Access
NVIDIA shipped four checkpoints -- an NVFP4-quantized build, a BF16 post-trained instruct model, a BF16 base model, and a GenRM reward-model variant -- with day-zero availability spanning more than twenty-five cloud and inference platforms, including OpenRouter, NVIDIA NIM, Hugging Face, Perplexity, Together AI, and Amazon SageMaker JumpStart.
API pricing on DeepInfra sits at $0.37/M input and $1.08/M output tokens -- competitive for a model at this capability tier. Self-hosting the full BF16 weights requires multi-GPU configurations (8xH100 or equivalent), but the NVFP4 checkpoint offers a more practical path for teams with Blackwell infrastructure. The model ships under the Linux Foundation's OpenMDW-1.1 license, which permits commercial use, modification, and redistribution.
For the open-weight AI ecosystem, AA-Briefcase is a useful forcing function: it makes it harder to claim agentic capability without demonstrating it on tasks that look like actual work. And for Nemotron 3 Ultra, a strong showing on a benchmark built around long-horizon, multi-file, multi-week projects is exactly the kind of validation that matters for teams considering it as the reasoning backbone of a production agent pipeline.