Unsloth Shrinks Qwen3.8-Flash-Next to 74.5 GB for Local Coding Agents

Unsloth has released new GGUF quantizations for Qwen3.8 GGUF files using its Dynamic V3.1 recipe. GGUF packages model weights for local inference through runtimes such as llama.cpp and Ollama. The smallest release occupies 74.5 GB, bringing a model with roughly 355 GB of BF16 weights within reach of a single high-memory workstation.

Unsloth reports a 15% accuracy improvement over competing quantizations on Terminal Bench and DeepSWE at a 256K-token context. The company has not specified in the supplied results whether that figure represents a relative gain or percentage-point increase, so direct comparisons require the underlying per-quant benchmark tables. The claim matters most for coding agents, where small prediction errors can compound across long tool-driven tasks.

One model, several parameter pools

Qwen describes the main model as a 125-billion-parameter mixture of experts, or MoE, that activates about 6 billion parameters for each token. Separate components add a 51-billion-parameter n-gram embedding bank and a 4-billion-parameter multi-token prediction head.

• Core stack: 48 layers arranged as 12 groups of three Gated DeltaNet blocks followed by one Qwen Sparse Attention block.
• Expert routing: Each MoE layer contains 512 experts and activates 10 routed experts plus one shared expert per token.
• Context: The native limit is 262,144 tokens. RoPE scaling can extend it toward one million tokens, with additional memory use and possible quality changes.

The n-gram bank stores representations for short token sequences in lookup tables, adding model capacity with less arithmetic than a conventional dense layer. An SSD-backed memory map can page portions of those tables into memory on demand. That approach reduces resident memory when the operating system can evict unused pages, although frequent random reads can make storage latency a bottleneck.

Quality falls unevenly

Unsloth compared each quantized model with the BF16 version using Kullback-Leibler divergence, which measures the difference between two token-probability distributions. Lower mean KLD indicates a closer match. The reported top-1 figure measures how often the quantized model chooses the same highest-probability token as BF16; task pass rates require separate benchmark runs.

The 74.5 GB release is about 79% smaller than the 355 GB BF16 model. Its “1-bit” label describes the most aggressively compressed weights, while Unsloth keeps the specialized PLE and n-gram components at a 4-bit floor because those lookup-heavy layers proved more sensitive to quantization.

File size is only the floor

Model files do not account for the runtime, temporary buffers, recurrent state, KV cache, or operating-system memory. Longer contexts increase those costs, so fitting a GGUF on disk does not guarantee that the same configuration will fit in memory.

• 96 GB systems: UD-Q2_K_XL leaves about 17 GB before runtime overhead. The 90 GB Q3 release leaves little room for long contexts.
• 128 GB systems: UD-Q3_K_XL provides a safer margin. The 111.3 GB Q4 release may fit with a modest context and memory-efficient cache settings.
• 192 GB systems: Q8_0 nearly consumes the entire capacity before overhead, so it generally requires additional memory or partial SSD-backed mapping.

Unified-memory Macs avoid copying weights between separate system RAM and VRAM pools, while large-RAM Linux systems can keep quantized weights in ordinary memory. In both cases, usable context length depends on the selected cache format, batch size, runtime implementation, and amount of memory reserved by the operating system.

Sparse routing changes the hardware equation

CPU inference is practical because each token uses a small subset of the model’s experts, reducing arithmetic relative to a dense model of similar total size. The complete weight set still has to reside in memory or be paged from storage, making memory bandwidth and expert-access patterns central to performance. GPUs retain substantial compute and bandwidth advantages, while unified memory and fast system RAM make local execution possible without enough dedicated VRAM for the full file.

Multi-token prediction, or MTP, lets the auxiliary head draft several future tokens and verify them together. Unsloth reports a 1.3 to 1.7 times speedup without an accuracy penalty, including roughly 170 tokens per second on one RTX 6000 Pro compared with 100 tokens per second when MTP is disabled. Those numbers depend on quantization, context length, batch size, and runtime support.

Two benchmark claims, two meanings

Qwen reports full-precision scores of 58.7 on DeepSWE 1.1, 62.5 on SWE-bench Pro, and 81.0 on SWE-bench Multilingual. Its published comparison places the model above Claude Opus 4.6 on most coding evaluations in that suite.

Those scores describe the full-precision model. Unsloth’s 15% claim compares its quantizations with other compressed releases, while the KLD table measures output similarity to BF16. Reproducible deployment comparisons need the same quant level, runtime, prompt template, context length, sampling settings, and benchmark harness.

Start with the runtime and memory budget

A current Ollama build can fetch a quant directly from Hugging Face. This 2-bit example is the more practical starting point for a 96 GB machine:

ollama run hf.co/unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q2_K_XL

Recent llama.cpp builds can load the repository through the Hugging Face option. A shorter initial context leaves room to measure actual memory use before increasing it:

./llama-cli \
  -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q2_K_XL \
  -c 32768

The Unsloth documentation also covers its desktop application, which handles RAM offloading and enables MTP when supported. Older llama.cpp and Ollama builds may fail to recognize the architecture or may run without MTP, so the runtime version affects both compatibility and speed.

Settings that affect long agent runs

• Thinking mode: temperature=1.0, top_p=0.95, and top_k=20.
• Reasoning effort: Supported templates expose low, medium, and xhigh.
• Long-horizon agents: Unsloth recommends avoiding low reasoning effort because additional retries can increase total token use.
• Initial context: Start below the maximum, monitor peak memory and throughput, then raise the limit for workloads that need it.

Dynamic V3.1 gives 96 GB and 128 GB systems workable Q2 and Q3 options for local coding agents, with Q4 available when memory and context requirements allow. The reported 15% benchmark advantage remains a vendor claim pending reproducible per-quant results, but the published sizes and similarity measurements provide concrete guidance for choosing a build.