PrismML Shrinks Bonsai 27B to 3.9 GB so It Runs on an iPhone

PrismML just dropped Bonsai 27B, and the headline is hard to argue with: a 27-billion-parameter multimodal reasoning model that fits in 3.9 GB and runs on an iPhone. For context, the same model in standard 16-bit precision weighs 54 GB. Even a well-optimized 4-bit build clocks in at 18 GB. The Bonsai 27B 1-bit variant is smaller than most full-precision 2B models.

Two variants, one goal

The release ships two operating points, each targeting a different hardware tier:

• Ternary Bonsai 27B -- 5.9 GB, 1.71 effective bits per weight. Each weight is one of three values: {-1, 0, +1}, with a shared FP16 scale factor per group of 128 weights. Targets laptop-class deployment.
• 1-bit Bonsai 27B -- 3.9 GB, 1.125 effective bits per weight. Binary weights ({-1, +1} only), maximum compression. Targets phones.

Both variants are multimodal, with the vision tower shipping in a compact 4-bit form so on-device workflows can handle screenshots, documents, and camera input, not just text. Bonsai 27B carries a full 262K-token context and supports speculative decoding, compounding speed with lossless draft-and-verify acceleration. Everything is open-sourced under Apache 2.0.

How they got there

The key distinction from conventional quantization is that PrismML sidestepped the degradation problem by abandoning post-training quantization entirely -- Bonsai was built from the ground up as a native low-bit architecture. Standard quantization takes a trained full-precision model and rounds its weights down after the fact, which bleeds quality. PrismML instead uses Quantization-Aware Training (QAT), meaning the model learns with the ternary or binary constraint baked in from the start.

By enforcing ternary constraints in the forward pass and using full-precision gradients in the backward pass, the model learns feature representations optimized for ternary space, minimizing performance degradation even at extremely low bits. The full training algorithm is described as proprietary Caltech IP, but the paradigm aligns with established BitNet-style QAT research.

The architecture is derived from Qwen3.6 27B, a hybrid-attention causal language model with roughly 75% linear attention and 25% full attention, with the architecture itself left unchanged. The low-bit representation runs end-to-end across the language network -- embeddings, attention, MLPs, and the LM head -- with no higher-precision escape hatches. PrismML also trained custom DSpark speculative-decoding drafter layers on top, giving a lossless 1.37x decode speedup on the CUDA serving path.

What it's actually good at

Across a 15-benchmark suite spanning knowledge, reasoning, math, coding, instruction following, tool calling, and vision, Ternary Bonsai 27B retains 95% of the full-precision baseline, and 1-bit Bonsai 27B retains 90%. But the averages hide the most important story:

Math and coding are nearly untouched, and tool calling stays within a few points of full precision -- exactly the capabilities that agentic workloads depend on. The bigger drops show up in vision and instruction-following for the 1-bit variant, which is worth keeping in mind for use cases that lean heavily on those.

For comparison, the most aggressive conventional low-bit build of the same base model scores significantly lower than 1-bit Bonsai 27B while occupying 2.5x more memory. That's the real competitive bar: not full precision, but the best 4-bit quantized alternative.

The phone threshold is trickier than it sounds

Getting to 3.9 GB is only half the problem. A phone never exposes its full memory to an app -- a 12 GB iPhone offers about 6 GB for the model to use on-device, and the model shares that budget with its KV cache and activations. No conventional 27B build has ever cleared that bar. At 3.9 GB, 1-bit Bonsai 27B does, with room left over for context.

The 1-bit build holds a 100K-token context at 11.6 GB without any KV-cache compression -- a budget that fits mainstream laptops outright, while the conventional Q4_K_XL build needs roughly 25.6 GB before the first long document is loaded. Enabling the 4-bit KV cache shrinks that further: the 100K peak drops to about 6.8 GB, and the full 262K window fits in roughly 9.4 GB peak.

Why agentic workloads change the math

The framing PrismML is pushing isn't just "smaller model, same quality." It's about a fundamentally different deployment architecture for AI agents. Modern agentic systems don't make one model call -- they make hundreds, chaining reasoning steps, tool calls, and file reads in a loop.

Local execution changes the equation. When a model capable of sustained agentic work fits on the device, the agent can live inside the product: the marginal cost of a hundred-step loop is zero, and the user's data never leaves the machine. For cloud-based agentic pipelines, every step is a round-trip with latency and cost that compounds. Move the loop on-device, and both disappear.

This also opens a hybrid architecture pattern: route non-frontier and privacy-sensitive tasks to a capable local model, and reserve frontier cloud models for the hardest steps -- collapsing the cost-per-task of agentic systems.

Practical use cases

Bonsai 27B is not a lightweight chat model or a proof of compression in isolation -- it is built to do real work: reasoning through complex tasks, planning multi-step workflows, writing and debugging code, using tools, and supporting agentic execution locally. Concrete scenarios where this matters:

• On-device coding assistants that reason over your local codebase without sending files to the cloud
• Offline document analysis -- the 262K context window can hold long reports, contracts, or research papers
• Privacy-sensitive agentic loops -- medical, legal, or financial workflows where intermediate states can't leave the device
• Persistent background agents that run continuously on a laptop or phone without per-token API costs
• MCP-integrated tool use -- the demo shows Ternary Bonsai 27B running end-to-end agentic workflows with Hermes on an RTX 5090 and on an M5 Max

How to run it

The models are available now on Hugging Face in multiple formats: GGUF for llama.cpp (CUDA + Metal), MLX for Apple Silicon, and MLX Swift for iOS/macOS. PrismML forked llama.cpp to add the custom low-bit kernels needed for the hybrid-attention architecture. Getting started on CUDA looks like this:

# Clone PrismML's llama.cpp fork (includes Q1_0_g128 hybrid-attention kernels)
git clone https://github.com/PrismML-Eng/llama.cpp
cd llama.cpp
# Build with CUDA support
cmake -B build -DGGML_CUDA=ON && cmake --build build -j
# Download the 1-bit GGUF weights
hf download prism-ml/Bonsai-27B-gguf Bonsai-27B-Q1_0.gguf --local-dir .

The Ternary variant is also available free via the Together AI API for those who want to evaluate it before committing to local deployment. There's also a WebGPU demo that runs the 1-bit model directly in the browser. PrismML is offering a free, limited-time developer preview API for direct access.

What this means for the field

The implicit assumption for years has been that capability scales with model size, and model size scales with infrastructure. For years, the assumption has been that more capable models require larger infrastructure: more memory, more compute, more power, and more dependence on cloud inference. Bonsai 27B challenges that assumption directly -- it shows that the next innovation in AI is not only bigger models in larger data centers, but also more capable models that can run privately, locally, and continuously on the devices people already use.

PrismML is a startup that emerged from Caltech research with backing from Khosla Ventures, Cerberus, and Google. Their Bonsai family has been steadily pushing the intelligence-density frontier -- 1-bit Bonsai 8B, then Ternary Bonsai across 1.7B/4B/8B sizes, then a Bonsai Image 4B model for on-device image generation. The 27B release is the biggest jump yet, and PrismML says the methodology is architecture-agnostic, with larger models already in progress.

Speed numbers give a sense of what's practical today: Bonsai 27B reaches up to 163 tok/s in 1-bit and 134 tok/s in Ternary on an NVIDIA GeForce RTX 5090. On an M5 Max, it reaches up to 87 tok/s in 1-bit and 58 tok/s in Ternary. Those are usable speeds for interactive agentic workflows, not just batch processing. The question now is whether the broader ecosystem -- inference runtimes, hardware vendors, and application developers -- moves fast enough to meet the model where it is.