Edge0 streams a 35B MoE from SSD on a Mac mini

The open-source Edge0 repository runs a 35 billion-parameter mixture-of-experts model on Apple Silicon by using SSD storage as a managed memory tier. Expert weights load on demand, while a small predictor anticipates the next experts and overlaps storage reads with model computation.

That design lowers the 35B tier’s reported peak active memory to 2.9 GiB on a Mac mini M4 Pro with 24 GB of unified memory. The full checkpoint still occupies about 23 GB of storage, and decode performance depends on SSD speed and cache state.

Experts on disk, one step ahead

Mixture-of-experts models contain many specialized parameter blocks called experts, but each token activates only a small subset. Compute scales with the selected experts, while conventional inference still needs enough memory for the full set of weights. Edge0 leaves those weights in memory-mapped files and asks the operating system to fetch the required pages.

• SSD expert offload: Memory-mapped expert files keep the active working set in memory while the complete expert set remains on storage.
• Prerouter: A trained prediction head selects likely expert candidates one step early, allowing their weights to load during the current forward pass. Edge0 reports up to 59% higher decode throughput than on-demand loading in its tests.
• Recover-LoRA: The 4-bit integer base weights remain frozen while low-rank adapters learn from a higher-precision teacher, recovering part of the quality lost during quantization.