Unsloth's Qwen3.6-35B Hits 17,561 Tokens per Second on a Single GPU

Unsloth's Qwen3.6-35B-A3B-NVFP4-Fast is a quantized checkpoint of Alibaba's Qwen3.6 MoE model that pushes throughput numbers most people associate with multi-GPU clusters onto a single 32GB GPU. Unsloth released NVFP4-quantized versions of Qwen3.6 -- both the 27B dense and the 35B-A3B MoE -- with a headline claim of 2.5x faster inference, with the 35B-A3B variant hitting 17,561 tokens/second on a B200. That kind of throughput on a single GPU changes the economics of running frontier-class models.

What NVFP4 actually is

Most quantization formats like INT4 or GGUF Q4 store weights in 4-bit integers but still have to dequantize back to 16-bit floats before any computation happens -- a hidden tax on every forward pass. NVFP4 is NVIDIA's 4-bit floating-point format for Blackwell GPUs. It uses two-level scaling -- an FP8 micro-block scale per 16 values plus an FP32 tensor-level scale -- achieving roughly 1.8x memory reduction versus FP8.

The key insight is that Blackwell Tensor Cores are designed to handle NVFP4 operations natively, so tensors stay in their compact 4-bit format throughout the entire forward pass. No dequantization, no hidden overhead. In plain terms: INT4 pretends to be 4-bit but secretly does 16-bit math. NVFP4 on a Blackwell GPU actually does 4-bit math end-to-end, giving 2-3x higher arithmetic throughput compared to FP8.

What Unsloth added on top

Unsloth applied their "dynamic" method to NVFP4 using W4A4 (weights and activations both at 4-bit) vs the standard W4A16 approach. They released two 35B-A3B versions -- NVFP4 (1.56x faster) and NVFP4-Fast (1.79x faster). Fast uses pure W4A4, while the non-Fast version uses a mixture to stay a bit more accurate.

The checkpoint is calibrated on a mixture of Unsloth's own dataset and the UltraChat dataset. Calibration matters a lot for quantization -- it determines which layers get quantized aggressively and which stay at higher precision. Unsloth also verified that output lengths across benchmarks are comparable, confirming the quants do not think for longer to compensate -- because if it's 2x faster but thinks 2x more, that's useless.