OrcaRouter Shrinks Flint's 40B Coding Model to Fit on Consumer GPUs
OrcaRouter shipped GGUF builds of an abliterated 40B MoE coding model that run on a 16GB or 32GB GPU with 262K context.
- OrcaRouter released Flint-abliterated-GGUF, an uncensored 40B MoE with 262K context.
- Two builds: 12 GB for 16 GB GPUs, 24 GB for 32 GB GPUs, Apache 2.0.
- Harmful-prompt refusal drops from 86% to 10% with MMLU barely moving.
- Mix-24G beats Unsloth UD-Q4_K_XL on KLD while being 1.1 GB smaller.
- Uses measured-KLD knapsack search plus GPTQ-style error-compensated rounding into stock llama.cpp types.
- Needs llama.cpp master with PR #30928; older Ollama and LM Studio will not load it.
OrcaRouter’s GGUF release packages RecursiveSI’s Flint into two quantized files targeting 16 GB and 32 GB GPUs. The builds also modify the model to reduce refusals, use standard llama.cpp data types, and retain Flint’s roughly 262,000-token context architecture.
GGUF is the model format used by llama.cpp and applications built on it. Quantization compresses weights into lower-precision representations, reducing memory use at the cost of some output fidelity. These builds matter because they make a 39.8-billion-parameter coding model available on a single consumer GPU without custom inference kernels.
Two files, different trade-offs
| Build | File size | Advertised GPU tier | Average expert precision | Top-1 match with BF16 | Perplexity change |
|---|---|---|---|---|---|
| Mix-12G | 11.99 GB | 16 GB | 2.27 bits | 88.13% | +20.0% |
| Mix-24G | 23.99 GB | 32 GB | 4.65 bits | 96.45% | +1.1% |
Flint uses a sparse Mixture-of-Experts architecture with 256 routed experts, top-6 routing, and one shared expert. Each token activates about 2.5 billion parameters, reducing computation relative to a dense 39.8B model, while GPU memory must still store the full set of experts.
The attention system combines sliding-window and full attention with learned attention sinks, which provide stable targets over long sequences. OrcaRouter reports a KV cache footprint of about 1.9 GB at 64K tokens and 3.7 GB at 128K tokens. Runtime buffers, the CUDA context, and the KV cache consume memory beyond the GGUF file size, so usable context depends on the GPU, backend, and offload configuration.
A one-vector edit cuts refusal rates
Abliteration identifies a direction in a model’s hidden states associated with refusals, then removes that component from selected weights. OrcaRouter calculated one difference-of-means direction from layer 17 and orthogonalized it out of the attention output projections, routed-expert down projections, and token embeddings. The edit was applied in BF16 before quantization.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
