Skip to main content
Back

NVIDIA's Puzzle-75B-A9B Squeezes a 120B Model Into 4x the Throughput

NVIDIA compresses its 120B flagship model to 75B using Iterative Puzzle, doubling server throughput while preserving near-identical accuracy across reasoning, coding, and long-context tasks.

  • NVIDIA released Nemotron-Labs-3-Puzzle-75B-A9B, compressing its 120B flagship to 75B total / 9.3B active parameters.
  • Achieves ~2x server throughput on an 8xB200 node at matched user throughput; with MTP enabled, reaches 4.63x vs. Super without MTP.
  • On a single H100 at 1M-token context, concurrency jumps from 1 to 8 simultaneous requests due to the smaller 44.5 GB weight footprint.
  • Uses a new Iterative Puzzle framework: alternating rounds of structured pruning and knowledge distillation recovery, targeting MoE channels, active expert count, and Mamba SSM state size.
  • Benchmarks stay within ~1–2 points of the parent on reasoning, coding, and long-context tasks; largest drops are on Arena-Hard-V2 (~4 pts) and SWE-Bench (~2.6 pts).
  • Available now on Hugging Face in BF16, FP8 (Hopper), and NVFP4 (Blackwell) under the OpenMDW-1.1 commercial license.

Nemotron-Labs-3-Puzzle-75B-A9B is NVIDIA's answer to a question that every production ML team eventually faces: what do you do when your best model is too expensive to serve at scale? Rather than training a smaller model from scratch, NVIDIA took their flagship Nemotron-3-Super-120B and surgically compressed it down to 75B total / 9.3B active parameters using a new framework called Iterative Puzzle. The result: roughly 2x server throughput on Blackwell hardware with benchmarks that stay within a few points of the original across nearly every task.

The problem that needed solving

Large hybrid MoE models like Nemotron-3-Super are accurate but expensive to serve. Their active parameters, KV cache, and Mamba state cap how many users a node can hold at a given per-user token rate. This becomes especially painful at long contexts. On a single H100 at 1M context, the binding constraint flips from compute to memory. Super's NVFP4 weights occupy about 70 GB of the 80 GB HBM budget. Each 1M-token request adds about 4 GB of KV cache. Effective concurrency is therefore 1. One user at a time, on a $30,000+ GPU, for a million-token context window. That's the problem Puzzle-75B-A9B was built to fix.

What Iterative Puzzle actually does

The core technique is a form of hardware-aware neural architecture search (NAS) , a method that automatically finds a smaller architecture optimized for a specific deployment target, rather than just uniformly shrinking the model. The key word is iterative. Standard one-shot compression scores each layer independently and prunes everything at once, which misses inter-layer dependencies. Iterative Puzzle instead alternates between moderate pruning steps and short knowledge distillation recovery phases, recomputing layer importance scores after each round.

Compression targets three architectural axes simultaneously:

  • Heterogeneous MoE channel pruning: The internal width of each routed expert (the intermediate dimension) is pruned non-uniformly across layers, from a fixed 2688 down to a layer-dependent range of 1280–2688. Layers that matter more keep more capacity.
  • Heterogeneous top-k reduction: Each token is routed to fewer experts per layer. The parent model used k=22 experts per token; Puzzle reduces this to a layer-dependent range of 4–18, cutting active parameters and improving throughput in compute-bound scenarios like prefill.
  • Mamba SSM state pruning: The Mamba state size (the recurrent memory buffer used by Mamba layers) is reduced from 128 to 96 channels uniformly. This produces a 1.2x–1.3x speedup of the SSM kernel during the decode stage for batch sizes between 8 and 512.