Skip to main content
Back

LocalAI's Qwopus3.6 Runs 30% Faster With One File Instead of Two Models

The LocalAI team shipped GGUFs of a 35B mixture-of-experts fine-tune that bundle a draft head inside one file for self-speculative decoding.

  • LocalAI shipped APEX-MTP GGUFs of Qwopus3.6-35B-A3B, a Qwen3.6 MoE fine-tune with 3B active parameters.
  • Files bundle a multi-token prediction head for self-speculative decoding, no separate draft model required.
  • Enabled via one flag in llama.cpp: --draft-mtp, thanks to PR #22673.
  • Reported speedups of 20-50% on long reasoning and coding outputs; short chats can be slower.
  • MTP overhead is only about 2.5% larger file size versus non-MTP quants.
  • Apache 2.0 licensed, free, and compatible with Ollama, LM Studio, and Docker Model Runner.

Running a 35B model on a workstation used to mean waiting through slow token streams or juggling a second draft model to speed things up. A new release from the LocalAI team collapses that setup into a single file. The Qwopus3.6-35B-A3B APEX-MTP GGUF drop packages a mixture-of-experts model together with its own draft predictor, so a recent llama.cpp build can do speculative decoding without loading a second model into memory.

What's actually in the file

The base is a community fine-tune of Qwen3.6 35B-A3B, with Jackrong/Qwopus3.6-35B-A3B-v1 serving as the adapter on top. The architecture has 40 trunk layers plus 1 MTP layer, 256 routed experts plus 1 shared expert with 8 active per token, and a hidden size of 2048. Only about 3B parameters fire per token even though the total sits at 35B, which is what makes it feasible to serve locally.

The novel piece here is APEX, short for Adaptive Precision for EXpert Models. It's a MoE-aware mixed-precision quantization scheme where different tensor roles get different bit widths:

  • Routed experts get the most aggressive compression