NVIDIA's Nemotron 3.5 Super VL Beats a 550B Model With 120B Parameters
NVIDIA's new open 120B vision-language MoE activates just 12B parameters per token, handles 1M context, and runs on a single DGX Spark at NVFP4.
- NVIDIA released Nemotron 3.5 Super VL, a 120B/12B-active open VLM with 1M context.
- Hybrid Mamba-2 + attention + MoE backbone with 512 experts and a C-RADIO vision encoder.
- NVFP4 build runs on 1x DGX Spark, 1x H200, 2x H100, or 1x GB200.
- Beats Nemotron 3 Ultra on most benchmarks: SWE-Bench Pro 51.78, GPQA 89.52, LiveCodeBench 88.22.
- Native multi-token prediction speculative decoding, roughly 630 output tokens/sec at 100K context.
- Open weights, datasets, recipes, and technical report under OpenMDW-1.1 license.
NVIDIA has released Nemotron 3.5 Super VL, an open-weight model that processes text, images, documents, and short videos from one checkpoint. It can reason, write code, call tools, and work with contexts of up to one million tokens. The model ships under the custom OpenMDW-1.1 license.
The checkpoint contains 120 billion parameters and activates 12 billion for each token. NVIDIA reports that it surpasses the 550B Nemotron 3 Ultra on most listed benchmarks while containing roughly 22% as many total parameters. For agent developers, one checkpoint can cover roles often split between a text model and a vision-language model.
Sparse routing cuts per-token work
Nemotron 3.5 Super VL combines a mixture-of-experts backbone with interleaved Mamba-2 and attention layers. Each MoE layer has 512 routed experts, while a C-RADIO vision encoder and connector translate visual inputs into representations the language backbone can process.
Mamba-2 blocks process long sequences with lower computational growth than full attention. Attention layers handle retrieval and relationships that require precise token-to-token lookup. Sparse expert routing limits per-token computation to 12B active parameters, although deployment still requires enough memory for the complete weights and the key-value cache.
- Multi-token prediction: A shared head predicts several upcoming tokens, allowing the model’s own head to serve as the drafter during speculative decoding.
- NVFP4 quantization: Blackwell hardware can use 4-bit weights and activations. Other supported hardware uses 4-bit weights with 16-bit activations. NVIDIA says quantization-aware distillation preserves quality close to the BF16 checkpoint and supports an NVFP4 key-value cache.
- Shared context window: Prompts, images, conversation history, tool results, and generated output consume the same one-million-token budget.
NVFP4 brings serving to one system
NVIDIA lists the following minimum hardware configurations for its published checkpoints:
| Checkpoint | Numeric format | Listed hardware |
|---|---|---|
| BF16 | 16-bit weights and activations | 4× H100 or 2× GB200 |
| NVFP4 checkpoint | W4A4 on Blackwell; W4A16 elsewhere |
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
