Offrail 27B fits speculative decoding into a 6.3 GB GGUF

Developer sudoingX has released Offrail 27B, a reduced-refusal 27-billion-parameter model packaged to fit an 8 GB gaming GPU at practical context lengths. The model card reports generation at roughly 50 tokens per second on an RTX 3060 12GB by combining ternary quantization with a multi-token prediction head for speculative decoding. GGUF packages model tensors and metadata for llama.cpp-family runtimes.

The release integrates three existing pieces: PrismML’s ternary-compressed Qwen3.5 27B model, OS-Software’s Heretic weight edits, and Qwen’s multi-token prediction head. Its contribution lies in making those components work together inside one deployable file. The performance figures come from the release author’s tests and have not been independently reproduced.

A three-project stack

Offrail starts with OS-Software’s Heretic GGUF of Ternary Bonsai 2 27B, itself a ternary-compressed version of Qwen3.5 27B. Its 5.95 GB PTQ1_0 body stores weights at roughly 1.75 bits per parameter. A rank-128 Heretic LoRA, covering layers 27 through 44, has been merged into the ternary codes while preserving the quantization block scales.

Qwen’s multi-token prediction, or MTP, head is appended as transformer block 64. The head proposes future tokens from the model’s hidden state, then the main model verifies those proposals together. Accepted drafts reduce the number of full decoding passes without replacing the target model’s final token decisions.

Two files trade memory for speed

• Offrail-27B-12G.gguf is 7.0 GB and includes a separate token-embedding table for the draft head.
• Offrail-27B-8G.gguf is 6.3 GB and reuses the trunk’s embedding table through a Hadamard inverse fix, saving 682 MiB.

The smaller build gives up about 1.2 tokens per second in the reported benchmark, generating 49.6 tokens per second compared with 50.8 for the 12G file.

One extra block adds up to 23%

Author-run benchmarks used an RTX 3060 12GB with flash attention and a q4_0 KV cache. Flash attention reduces attention-related memory traffic, while the quantized KV cache compresses the stored attention state required for long contexts.