Skip to main content
Back

LocalAI's Qwen3.6 Runs 2x Faster on a Single Consumer GPU

LocalAI's APEX-MTP quants bundle Qwen3.6-35B's speculative decoding head into a single GGUF, delivering self-drafting speedups with zero extra models.

  • LocalAI released APEX-MTP GGUF quants of Qwen3.6-35B-A3B with self-speculative decoding built in
  • Multi-token prediction head bundled in a single file, no separate draft model needed
  • Enabled via --draft-mtp flag in mainline llama.cpp (PR #22673)
  • APEX applies role-aware mixed precision: routed experts compressed hardest, shared experts kept high
  • 256 routed experts, 8 active per token, 40 trunk layers plus 1 MTP layer, Apache 2.0 license
  • Reported gains up to ~78% tok/s on A10G; best on code, JSON, and agent workloads

Running a 35B mixture-of-experts model on a single consumer GPU used to mean picking between speed and simplicity. A new release from the LocalAI team collapses that trade-off: Qwen3.6-35B-A3B-APEX-MTP-GGUF, a mixed-precision quantization of Qwen's latest MoE that carries its own speculative decoding drafter inside a single file.

The mechanism is called multi-token prediction, or MTP. Conventional speculative decoding pairs a big target model with a small draft model that guesses several tokens ahead, which the big model then verifies in parallel. MTP bakes that draft path into the model itself. Rather than loading a separate checkpoint to guess the next few tokens, the main model emits multiple candidate tokens per forward pass. No second file to download, no tokenizer mismatch headaches, no juggling two models in VRAM.

One file, self-drafting

These GGUFs bundle the MTP head alongside the trunk in a single file, courtesy of llama.cpp PR #22673. Once you're on a recent llama.cpp build, spinning up self-speculative decoding is a one-liner:

llama-server -m Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf --draft-mtp

The MTP path is now part of mainline llama.cpp's --spec-type system alongside draft-simple, EAGLE-3, and ngram-cache variants, so no custom fork is required. Because the drafter shares the trunk's embedding table, that extra token-prediction capability adds only about 2.5% to file size versus the non-MTP quant.