Skip to main content
Back

Flash-MoE Runs a 397B AI Model on a MacBook With 48GB RAM

A pure C and Metal engine streams a 397B parameter Qwen model from SSD on a 48GB MacBook, hitting 4.4 tokens per second with tool calling intact.

Pro

  • Flash-MoE runs a 397B parameter Qwen MoE model on a 48GB MacBook at 4.4 tokens/sec.
  • Pure C, Objective-C and Metal shaders, no Python or ML frameworks anywhere in the stack.
  • 209GB of expert weights stream on demand from SSD via parallel pread() calls.
  • Custom FMA-optimized 4-bit dequant kernel adds 12% throughput over the naive formulation.
  • OS page cache beats every custom LRU tested, hitting ~71% naturally at zero overhead.
  • 2-bit quantization is faster but breaks JSON output and tool calling reliability.

Running frontier-scale language models on a laptop was supposed to be impossible. A new open-source project called Flash-MoE disagrees, and it does so without a single line of Python or PyTorch. The engine is a pure C/Metal inference stack that runs Qwen3.5-397B-A17B, a 397 billion parameter Mixture-of-Experts model, on a MacBook Pro with 48GB RAM at 4.4+ tokens/second with production-quality output including tool calling. The entire 209GB model streams from SSD through a custom Metal compute pipeline.

The trick: only load what you need, when you need it

A dense 397B model would never fit in 48GB of unified memory. But Mixture-of-Experts models don't use every parameter on every token. The model has 60 transformer layers: 45 GatedDeltaNet (linear attention) + 15 standard full attention. Each layer has 512 experts, of which K=4 are activated per token (plus one shared expert). That means only a tiny slice of the network is actually needed per forward pass, and Flash-MoE exploits this ruthlessly.

Expert weights (209GB at 4-bit) are read from NVMe SSD on demand via parallel pread() with GCD dispatch groups. Only the K=4 active experts per layer are loaded (~6.75MB each). The OS page cache manages caching, no custom cache needed ("Trust the OS" principle). Inspired by Apple's "LLM in a Flash" paper. The math checks out: instead of holding hundreds of gigabytes in RAM, the runtime reads roughly 27MB of expert data per layer from a fast SSD.

Numbers that matter

The benchmark table on the repo shows the tradeoff between speed and output quality clearly:

Configurationtok/sQualityNotes
4-bit experts, FMA kernel4.36ExcellentFull tool calling, 209GB on disk