Colibri Runs a 744B Model on a Laptop With No GPU

A hobby project that promised to run a 744-billion-parameter model on a laptop has become one of the most-starred inference engines on GitHub, sitting at over 25,000 stars. Colibri is a pure C inference runtime that treats storage, RAM, and VRAM as a single memory hierarchy, streaming Mixture-of-Experts weights from an SSD on demand so a giant model no longer needs giant hardware.

The pitch is blunt: GLM-5.2 int4, a 744B MoE with 40B active parameters, running on a 25GB RAM machine with no GPU. That configuration produces roughly 0.05 to 0.1 tokens per second, which nobody would call fast, but the same engine also scales up to 5.8 to 6.8 tok/s on six RTX 5090s with the entire expert set resident in VRAM. Same C file, same int4 container, only the placement changes.

Parameters as JIT-compiled code paths

The trick behind Colibri is a re-framing of what parameters are. The README puts it clearly: a 744B MoE only activates ~40B parameters per token, and only about 11 GB of those routed experts actually change between tokens. Rather than loading all 744B into memory, the engine treats them like a JIT compiler treats code paths, staging only the hot ones just in time.

Concretely, this splits the model in two:

• The dense part (attention, shared experts, embeddings, roughly 17B params) stays resident in RAM at int4, about 9.9 GB.
• The 19,456 routed experts (75 MoE layers times 256 experts, plus an MTP head, roughly 19 MB each at int4) live on disk as a ~370 GB blob and are streamed on demand.