OrcaRouter's OrcaSAQ2 Squeezes a 27B Coding Agent Into 16 GB GPUs
OrcaRouter shrinks Qwen3.8-27B from 54GB to 12.3GB using 3-bit mixed-precision quantization, keeping 70% SWE-bench Verified on a single 16GB GPU.
- OrcaRouter released OrcaSAQ2 27B, a 3.21 bpw quant of Qwen3.8-27B at 12.3GB.
- Scores 70.0 SWE-bench Verified and 58.4 Terminal-Bench 2.1, matching Claude Sonnet 4.6 + Claude Code.
- Only +0.02% perplexity vs BF16, with 93.2% token-level top-1 agreement and 0.031 mean KLD.
- Fits on a single 16GB GPU, hitting 90.1 tok/s single-stream with MTP speculative decoding.
- Ships with 262K context, tool calling, thinking mode, and vLLM serving under Apache-2.0.
- Quantization method is proprietary and undisclosed; requires OrcaRouter's custom vLLM kernel to run.
OrcaSAQ2 27B squeezes an agent model onto 16 GB GPUs
OrcaRouter has released OrcaSAQ2 27B, a mixed-precision quantization of Qwen3.8-27B aimed at local coding and tool-using agents. According to the model card, the checkpoint shrinks from 54 GB in BF16 to 12.3 GB while retaining the base model’s configured 262K-token context window.
Quantization stores model weights at lower precision to reduce memory use and often increase inference speed. Agent workloads make the tradeoff harder because one incorrect tool name, file path, or command argument can alter the environment and derail later steps. OrcaRouter’s central claim is that its 3.21-bit-per-weight build preserves enough behavioral fidelity for long, tool-driven tasks.
The 16 GB claim, unpacked
| Specification | BF16 base | OrcaSAQ2 |
|---|---|---|
| Checkpoint size | 54 GB | 12.3 GB |
| Storage reduction | None | 77.2% |
| Average precision | 16 bits | 3.21 bits per weight |
| Configured context | 262K tokens | 262K tokens |
Bits per weight is an average across the checkpoint. A mixed-precision quantizer can retain more bits for sensitive weights and use fewer elsewhere, while metadata and packing add storage beyond the raw weight calculation.
Runtime memory also includes activations, CUDA kernels, framework overhead, and the key-value cache used to retain context. The 12.3 GB checkpoint fits within a 16 GB card, but the full 262K-token window does not fit there under typical serving conditions. OrcaRouter suggests starting around 32K tokens for interactive use on a 16 GB GPU, then adjusting context and concurrency from measured memory consumption.
WikiText barely moves
| Fidelity metric | Reported result |
|---|---|
| BF16 WikiText-2 perplexity | 5.6468 |
| OrcaSAQ2 WikiText-2 perplexity | 5.6482 |
| Relative perplexity change | +0.02% |
| Token-level top-1 agreement | 93.2% |
| Mean KL divergence | 0.031 |
Perplexity measures how well a model predicts a text corpus. It can reveal broad degradation after quantization, but it does not show whether the model still selects valid tools, emits correct arguments, or recovers from errors during a multi-step task.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
