Skip to main content
Back

LocalAI Packs Self-Speculative Decoding Into One Carnice-Qwen3.6 File

A community contributor packaged samuelcardillo's 35B mixture-of-experts model into GGUFs that bundle a multi-token prediction head, unlocking self-speculative decoding from a single file.

  • mudler published GGUFs of samuelcardillo's Carnice-Qwen3.6 35B MoE with the MTP head bundled inside.
  • Enables self-speculative decoding via --draft-mtp in llama.cpp, with no separate draft model.
  • Requires llama.cpp PR #22673 (commit 255582687 or newer) to activate.
  • MTP head quantized to Q8_0 on most tiers, adding only about 2.5% file size.
  • APEX applies mixed precision by tensor role: routed experts compressed hardest, shared experts kept high.
  • Consumer Ampere with Q4 targets may see no throughput gain; server GPUs benefit more.

Speculative decoding usually asks you to run two models side by side: a big target and a smaller draft that guesses ahead. A new set of quantized GGUFs on Hugging Face collapses that setup into one file. Community contributor mudler, working under the LocalAI project's APEX quantization effort, has published Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-GGUF, a quantized build of samuelcardillo/Carnice-Qwen3.6-MoE-35B-A3B that ships the model's multi-token prediction head inside the same GGUF as the trunk.

One file, no draft model required

Multi-token prediction, or MTP, is a small extra transformer block trained to guess the next token cheaply. During inference the main model verifies that guess, and when the draft is correct you effectively get two tokens for the cost of one forward pass. Traditionally in llama.cpp you had to point --model-draft at a completely separate smaller model, matching tokenizers and juggling extra memory.

Thanks to llama.cpp PR #22673, the MTP head can now live in the same file as the main weights. With a recent llama.cpp (commit 255582687 or newer) you can enable self-speculative decoding using a single file, with no separate draft model: