Prism ML's Ternary-Bonsai-2 Shrinks a 9B AI Model to 2.42 GB
Prism ML squeezes a 9B vision-language model down to 2 GB with ternary weights, hybrid attention, and 262K context for on-device inference.
- Prism ML released Ternary-Bonsai-2-9B-gguf, a 9B VLM with ternary weights at ~2.42 GB.
- Compresses roughly 7.5x from FP16 using {-1, 0, +1} weights with FP16 group scales.
- Inherits Qwen3.5-9B hybrid attention and a 262,144-token context window.
- Ships three GGUF packings: PTQ1_0 (2.00 GB), PQ2_0 (2.42 GB default), Q2_0 (2.56 GB).
- Needs the PrismML llama.cpp fork for ternary kernels; Q2_0 runs upstream.
- Includes built-in MTP speculative decoding and optional DFlash2 drafter for faster inference.
Prism ML has released Ternary-Bonsai-2-9B, a quantized vision-language model derived from Qwen3.5-9B. Its default GGUF compresses the roughly 8.95-billion-parameter language backbone from about 17.9 GB at FP16 to 2.42 GB, while the vision components ship separately. The model repository showed more than 896,000 downloads and 3,700 likes at the time of writing, though Hugging Face totals can include repeated and automated downloads.
The smaller weight files expand the hardware available for local inference, including laptop GPUs, Apple silicon and CPU-only systems. Several constraints shape deployment: the smallest formats require Prism ML’s llama.cpp fork, runtime memory exceeds the GGUF file size, and using the full 262,144-token context window adds substantial memory and latency.
How 9B parameters fit into 2.42 GB
The language model stores its major weight matrices with ternary values from {-1, 0, +1}, paired with a shared FP16 scale for each group. The scheme covers embeddings, attention projections, MLP projections and the language-model head. Custom kernels read the packed representation during inference, avoiding a full FP16 copy of the weights.
A blockwise Hadamard rotation is folded into the stored weights, and the runtime applies the corresponding transform to activations. The rotation spreads large outlier values across dimensions, reducing the quantization error caused by mapping weights into three values.
| Packing | File size | Layout | Runtime support |
|---|---|---|---|
PTQ1_0 |
2.00 GB | Dense trits; smallest file | Prism ML fork |
PQ2_0 |
2.42 GB | Two-bit slots; default and fastest to unpack | Prism ML fork |
Q2_0 |
2.56 GB | Upstream-compatible layout | Prism ML fork and upstream llama.cpp |
The default PQ2_0 file is about 7.4 times smaller than the 17.9 GB FP16 language model. That ratio applies to the language weights on disk; it excludes the optional vision package, context state, compute buffers and runtime overhead.
A 262K window with practical limits
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
