Flash-MoE Runs a 397B AI Model on a MacBook With 48GB RAM
A pure C and Metal engine streams a 397B parameter Qwen model from SSD on a 48GB MacBook, hitting 4.4 tokens per second with tool calling intact.
ProTakeaways
- Flash-MoE runs a 397B parameter Qwen MoE model on a 48GB MacBook at 4.4 tokens/sec.
- Pure C, Objective-C and Metal shaders, no Python or ML frameworks anywhere in the stack.
- 209GB of expert weights stream on demand from SSD via parallel pread() calls.
- Custom FMA-optimized 4-bit dequant kernel adds 12% throughput over the naive formulation.
- OS page cache beats every custom LRU tested, hitting ~71% naturally at zero overhead.
- 2-bit quantization is faster but breaks JSON output and tool calling reliability.
Running frontier-scale language models on a laptop was supposed to be impossible. A new open-source project called Flash-MoE disagrees, and it does so without a single line of Python or PyTorch. The engine is a pure C/Metal inference stack that runs Qwen3.5-397B-A17B, a 397 billion parameter Mixture-of-Experts model, on a MacBook Pro with 48GB RAM at 4.4+ tokens/second with production-quality output including tool calling. The entire 209GB model streams from SSD through a custom Metal compute pipeline.
The trick: only load what you need, when you need it
A dense 397B model would never fit in 48GB of unified memory. But Mixture-of-Experts models don't use every parameter on every token. The model has 60 transformer layers: 45 GatedDeltaNet (linear attention) + 15 standard full attention. Each layer has 512 experts, of which K=4 are activated per token (plus one shared expert). That means only a tiny slice of the network is actually needed per forward pass, and Flash-MoE exploits this ruthlessly.
Expert weights (209GB at 4-bit) are read from NVMe SSD on demand via parallel pread() with GCD dispatch groups. Only the K=4 active experts per layer are loaded (~6.75MB each). The OS page cache manages caching, no custom cache needed ("Trust the OS" principle). Inspired by Apple's "LLM in a Flash" paper. The math checks out: instead of holding hundreds of gigabytes in RAM, the runtime reads roughly 27MB of expert data per layer from a fast SSD.
Numbers that matter
The benchmark table on the repo shows the tradeoff between speed and output quality clearly:
| Configuration | tok/s | Quality | Notes |
|---|---|---|---|
| 4-bit experts, FMA kernel | 4.36 | Excellent | Full tool calling, 209GB on disk |