Unsloth's Qwen3.6 NVFP4 Hits 17,561 Tokens per Second on a Single GPU
Unsloth's NVFP4 quants run Qwen3.6-27B on a 24GB GPU at 2.5x the speed, with MTP speculative decoding baked in
- Unsloth released NVFP4 quants for Qwen3.6-27B and 35B-A3B, claiming 2.5x faster inference on NVIDIA Blackwell GPUs.
- Qwen3.6-27B NVFP4 now fits in 24GB VRAM; 35B-A3B hits 17,561 tok/s on a B200 with MTP enabled.
- NVFP4 is NVIDIA's Blackwell-native 4-bit float format -- no dequantization overhead, 3.3x smaller than BF16.
- MTP speculative decoding is bundled directly into the checkpoint, adding 1.4-2.2x speed on top of NVFP4 gains.
- Full speed requires a Blackwell GPU (SM100+); on older hardware, you get memory savings but not the full throughput boost.
- Fixes included for tool calling, nested object parsing, developer role support, and a new Preserve Thinking mode for multi-turn accuracy.
Unsloth just released NVFP4-quantized versions of Qwen3.6 -- both the 27B dense model and the 35B-A3B MoE -- with a headline claim of 2.5x faster inference on NVIDIA GPUs. The 35B-A3B variant hits 17,561 tokens/second on a B200. That's not a typo. And the 27B now fits in 24GB of VRAM, which means a single consumer or prosumer GPU can run it.
What NVFP4 actually is
To understand why this matters, you need to know what NVFP4 is. Most quantization formats (INT4, GGUF Q4) store weights in 4-bit integers but still have to dequantize back to 16-bit floats before any computation happens. That dequantization step is a hidden tax on every forward pass.
NVFP4 is NVIDIA's 4-bit floating-point format for Blackwell GPUs. It uses two-level scaling -- an FP8 micro-block scale per 16 values plus an FP32 tensor-level scale -- achieving roughly 1.8x memory reduction versus FP8. More importantly, because Blackwell Tensor Cores are designed to handle NVFP4 operations directly, tensors stay in their compact 4-bit format throughout inference. There is no dequantization, and NVFP4 operations are hardware-accelerated.
In plain terms: INT4 pretends to be 4-bit but secretly does 16-bit math. NVFP4 on a Blackwell GPU actually does 4-bit math end-to-end. Compared to FP8, NVFP4 offers 2-3x higher arithmetic throughput and approximately 1.8x memory reduction.
What Unsloth added on top
The raw NVFP4 format is not new -- NVIDIA introduced it with the Blackwell architecture. What Unsloth brings is their calibration and quantization pipeline applied to it, plus a key integration: MTP tensors are directly integrated inside the NVFP4 quant, and both vLLM and SGLang work for this.