NVIDIA Squeezes a 74B Nemotron Bolt Into 24GB for Local AI Agents
NVIDIA squeezes a 74B parameter mixture-of-experts agent model into 24GB using 2-bit quantization, targeting local RTX hardware with full 256K context.
- NVIDIA Nemotron 3.5 Bolt: 74B MoE (9B active) compressed into a 24GB GGUF via 2-bit quantization.
- Compressed from 120B Super VL parent using Star Elastic router-guided architecture search plus 120B recovery tokens.
- Experts quantized to IQ2-XXS at 2.0625 bits with quantization-aware distillation; Mamba layers kept in NVFP4.
- Beats similarly-sized BF16 Qwen and Gemma on Terminal-Bench, GPQA, BrowseComp, and WebArena.
- Full 256K context, vision input, OpenAI-compatible server via llama.cpp, vLLM, Ollama, LM Studio.
- Community has run it on a single RTX 3060 12GB with expert offload at 24 tok/s decode.
NVIDIA has released Nemotron 3.5 Bolt, a 24GB GGUF model built for local text, image, video, and agent workloads through llama.cpp. The mixture-of-experts model contains 74 billion parameters but activates about 9 billion for each token. NVIDIA reports decoding speeds of at least 80 tokens per second when Bolt runs with its companion speculative-decoding model.
Mixture-of-experts models contain multiple specialist blocks while routing each token through a subset of them. This structure gives Bolt more stored capacity than a dense 9B model while limiting the computation required for each generated token. Its target workload is a local agent running beside browsers, development tools, and other desktop applications.
The 24GB figure covers the quantized model weights. Runtime memory must also hold the context, key-value cache, vision projector, and llama.cpp buffers. NVIDIA’s official configurations require about 28GB for a 64K-token context or 32GB for the full 262,144-token context. A 24GB GPU therefore needs partial CPU offload or a shorter context.
A 120B parent, cut to 24GB
NVIDIA derived Bolt from the 120B-parameter, 12B-active Super VL parent. The hybrid backbone combines Mamba-2 layers, attention layers, and mixture-of-experts blocks. A C-RADIO encoder and vision-language connector add image and video input.
Its Star Elastic compression system uses the parent model’s router to search for different expert counts and widths in each layer while enforcing a fixed memory budget. The resulting architecture activates nine experts per token, down from 22 in the parent.
Architecture search consumed 8 billion tokens at an 8K context length. Recovery training then used 72 billion tokens at 8K and 48 billion at 128K, producing 120 billion tokens of knowledge distillation after the search phase and 128 billion tokens across both phases.
Four post-training stages adapted the compressed language model for multimodal and agent work:
- Vision restoration: NVIDIA retrained the adapter with roughly 6.6 million image samples per epoch to reconnect the vision encoder.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
