Skip to main content
Back

Turboderp Squeezes Qwen3.8-27B Into 12 GB GPUs With EXL3

Turboderp's EXL3 quants of Qwen3.8-27B use trellis coding from QTIP to run a 27B dense model on a 12GB GPU with near-lossless quality.

  • Turboderp released EXL3 quants of Qwen3.8-27B from 1.4 to 6.0 bpw, Apache-2.0 licensed.
  • EXL3 uses trellis coding from QTIP, spreading rounding error across dimensions rather than per-weight.
  • At 4 bpw, EXL3 sits within 0.004 nats KL of the BF16 teacher, matching FP8 at half the size.
  • A 27B dense model runs on a 12 GB RTX 3060 at 25 tok/s with 64k context and vision.
  • Runs via ExLlamaV3 and TabbyAPI, with an OpenAI-compatible endpoint and MTP draft heads.
  • Self-quantization takes minutes to hours on a single RTX 4090, versus 720 GPU-hours for AQLM.

Turboderp, the developer behind ExLlamaV3, has released an extensive set of Qwen3.8-27B quants in the new EXL3 format. The collection spans 1.40 to 6.00 bits per weight, or bpw, and includes self-calibrated variants that quantize the model’s vision tower.

The lower-bitrate packs allow the 27-billion-parameter dense model to run within a 12 GB VRAM budget. Published measurements also show low divergence from the original BF16 model, giving developers another option when GGUF, AWQ, or FP8 cannot provide the desired balance of memory use, output fidelity, context length, and decoding speed.

A wider bitrate ladder

The repository contains standard EXL3 quants from 2.00 to 6.00 bpw and self-calibrated, or SC, variants down to 1.40 bpw. The SC branches use order-3 through order-6 Hadamard rotations, identified by labels such as H3 and H6. Vision-enabled branches require ExLlamaV3 1.4.4 or later.

The model card lists the quantized files under Apache-2.0. Developers redistributing applications or derived weights should also verify the terms attached to the upstream model and any bundled components.

Trellis paths replace isolated guesses

EXL3 derives from QTIP, short for Quantization with Trellis Information Processing. Many GGUF quantization types and AWQ approximate small groups of weights using a limited set of low-bit values. EXL3 jointly encodes vectors of weights as paths through a constrained codebook, allowing the optimizer to distribute quantization error across several dimensions.

Hadamard transforms spread large outlier values before encoding, making the transformed weights easier to represent at low bitrates. A fused Viterbi kernel then searches for an efficient path through the codebook. This structure becomes increasingly useful below 3 bpw, where independent or small-group approximations tend to lose accuracy quickly.