Unsloth's Qwen3.6 NVFP4 Hits 17,561 Tokens per Second on a Single GPU

Unsloth just released NVFP4-quantized versions of Qwen3.6 -- both the 27B dense model and the 35B-A3B MoE -- with a headline claim of 2.5x faster inference on NVIDIA GPUs. The 35B-A3B variant hits 17,561 tokens/second on a B200. That's not a typo. And the 27B now fits in 24GB of VRAM, which means a single consumer or prosumer GPU can run it.

What NVFP4 actually is

To understand why this matters, you need to know what NVFP4 is. Most quantization formats (INT4, GGUF Q4) store weights in 4-bit integers but still have to dequantize back to 16-bit floats before any computation happens. That dequantization step is a hidden tax on every forward pass.

NVFP4 is NVIDIA's 4-bit floating-point format for Blackwell GPUs. It uses two-level scaling -- an FP8 micro-block scale per 16 values plus an FP32 tensor-level scale -- achieving roughly 1.8x memory reduction versus FP8. More importantly, because Blackwell Tensor Cores are designed to handle NVFP4 operations directly, tensors stay in their compact 4-bit format throughout inference. There is no dequantization, and NVFP4 operations are hardware-accelerated.

In plain terms: INT4 pretends to be 4-bit but secretly does 16-bit math. NVFP4 on a Blackwell GPU actually does 4-bit math end-to-end. Compared to FP8, NVFP4 offers 2-3x higher arithmetic throughput and approximately 1.8x memory reduction.

What Unsloth added on top

The raw NVFP4 format is not new -- NVIDIA introduced it with the Blackwell architecture. What Unsloth brings is their calibration and quantization pipeline applied to it, plus a key integration: MTP tensors are directly integrated inside the NVFP4 quant, and both vLLM and SGLang work for this.

MTP (Multi-Token Prediction) is a form of speculative decoding where the model predicts several future tokens in parallel, then verifies them in a single pass. MTP enables 1.4-2.2x faster inference without accuracy loss. Bundling the MTP draft head directly into the NVFP4 checkpoint means this checkpoint includes the MTP module so it can act as its own speculative draft for faster decoding -- you just point vLLM at it and add the speculative config.

The calibration itself is also non-trivial. This is an Unsloth NVFP4 quantized checkpoint calibrated on the Hugging Face UltraChat dataset with sequences up to 16K context length and an approximately 2M-token calibration budget. That's a much larger and more realistic calibration set than most quantization pipelines use.

The numbers

Here's what the hardware picture looks like:

• Qwen3.6-27B NVFP4: fits in 24GB VRAM (single RTX 4090 / A10G / L4)
• Qwen3.6-35B-A3B NVFP4: 17,561 tok/s on a B200 with MTP enabled
• MTP speedup: 1.4x for dense models, 1.15-1.25x for MoE, with 83% token acceptance rate at 2 draft tokens
• Memory savings: NVFP4 reduces weight storage by approximately 3.3x versus BF16 and 1.5-1.8x versus FP8

The MTP acceptance rate is worth paying attention to. Unsloth's benchmarks show that using 2 draft tokens hits 83% acceptance, but jumping to 4 draft tokens drops acceptance to 50%, at which point the overhead of extra forward passes cancels out the gains. Two is the sweet spot.

The quantization quality story

Speed is only useful if accuracy holds. Unsloth's approach here is their Dynamic 2.0 quantization method, which they apply to the NVFP4 calibration. Instead of applying uniform quantization across all model layers, it analyzes each individual layer and picks the quantization type that minimizes accuracy loss for that specific layer.

Unsloth identified that using Wikipedia-only calibration data causes models to overfit on benchmark sets sharing the same source. They built Calibration_v3 and Calibration_v5, curated collections of over 1.5M tokens spanning dialogues, code snippets, and multilingual content.

They also fixed several practical issues that have plagued Qwen3.6 deployments:

• Tool calling: Improved parsing of nested objects to make tool calling succeed more reliably.
• Developer role support: Uploads now support the developer role for agentic coding tools like OpenAI Codex and OpenCode.
• Preserve Thinking: A new mode that carries the reasoning trace from previous turns in a conversation, trading token count for better multi-turn accuracy.
• CUDA warning: Do not use CUDA 13.2 -- it produces gibberish outputs. Use CUDA 13.3 or below.

Hardware requirements and constraints

With the introduction of NVIDIA Blackwell (B200) GPUs, NVFP4 benefits from native FP4 tensor cores, enabling true hardware-accelerated FP4 compute. That's the catch: on machines below SM100 (pre-Blackwell), vLLM will not run activation quantization -- only weight-only quantization. You still get the memory savings on older hardware, but not the full throughput multiplier. The 2.5x speed claim is a Blackwell number.

For teams without B200s, the GGUF variants via llama.cpp remain the practical path. Qwen3.6-27B runs on 18GB RAM setups and 35B-A3B runs on 22GB using the 4-bit GGUF quants, which work on any GPU or even CPU.

How to run it

vLLM is the recommended serving path for this NVFP4 checkpoint. The one-liner to get started with MTP enabled on the 35B-A3B:

vllm serve unsloth/Qwen3.6-35B-A3B-NVFP4 \
  --trust-remote-code --dtype bfloat16 --moe-backend marlin \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}'

SGLang also works for the 27B:

python -m sglang.launch_server \
  --model-path unsloth/Qwen3.6-27B-NVFP4 \
  --speculative-algo NEXTN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4

Both models are free and available now on Hugging Face. The models support a native context length of 262,144 tokens, extensible up to 1,010,000 tokens via YaRN. Qwen3.6 is a hybrid reasoning model, so you can toggle thinking mode on or off at serve time with --chat-template-kwargs '{"enable_thinking":false}'.

The bigger picture

This release is a preview of where local and on-prem inference is heading. The combination of NVFP4 + MTP speculative decoding stacks two independent speedups: one from doing less memory bandwidth work per token, and one from generating multiple tokens per forward pass. NVFP4 delivers near-baseline accuracy at large scale, making it well suited for frontier and MoE models.

For teams running agentic workloads -- coding agents, tool-calling pipelines, multi-step reasoning -- the throughput gains here are directly translatable to lower latency and cost. Qwen3.6 supports 256K context across 201 languages and excels in agentic coding, vision, and chat tasks. Getting that capability at 2.5x the speed, on hardware you already own, is the practical win.