Mirai Labs' Qwen3.8-27B Squeezes 128K Context Into 12 GB of VRAM

A community port now runs Mirai Labs’ 2.4-bit Qwen3.8-27B build in llama.cpp. Its 12 GB configuration uses 7.8 GB of VRAM for the model and a 128K-token context window, while keeping the 2.5 GB token embedding in system memory. The Hugging Face checkpoint copies Mirai’s compressed codes unchanged into GGUF, avoiding another lossy quantization pass.

How 27 billion parameters fit

The port starts with Mirai Labs’ source checkpoint, which uses the Mirai S codec to store weights at 2.4 bits each. The compressed weights occupy 8.45 GB, compared with roughly 56 GB for BF16 weights and about 17 GB for a conventional 4-bit GGUF before allocating the key-value cache used to retain context.

The release contains two files. An 11.2 GB GGUF holds the language model and its multi-token prediction block, which can accelerate generation through speculative decoding. At runtime, 7.8 GB of that file goes to the GPU and the 2.5 GB F16 token embedding remains in system RAM. An optional 0.93 GB vision encoder comes from the base Qwen3.8-27B model because Mirai’s checkpoint includes only the language model.

RTX 3090 results, with caveats

The port’s published benchmarks used one RTX 3090 capped at 300 W, with available VRAM restricted in software to emulate smaller cards. The measurements therefore show memory feasibility on the tested GPU, while actual 12 GB and 16 GB cards may differ in memory bandwidth, compute throughput, and power limits.