llama-modchip Squeezes DeepSeek-V4-Flash's 284B Model Into 64 GiB Without a GPU
A set of llama.cpp patches and custom quantizations lets a 284B DeepSeek MoE model run entirely in 64GB of system RAM with zero GPUs.
- llama-modchip runs 284B DeepSeek-V4-Flash on a 64GB CPU with zero GPUs at 9-14.7 tok/s decode.
- Custom Fractal-64 and SwiGLU-H2 quantization cuts the checkpoint from 76.87 GiB to 56.56 GiB losslessly.
- AVX-512 multi-column tiling kernels boost prefill 77% to 44.36 tok/s on 16-core Zen 5.
- Stockfish-PVS speculative drafter plus /dev/shm KV snapshots give 13ms Claude Code context restore.
- Verified on Zen 5, Threadripper, Apple M4, ARM Graviton; 30B MoEs hit 25-111 tok/s on CPU.
- Custom tensor type 40 requires the bundled patched llama.cpp binary; incompatible with vanilla Ollama/LM Studio.
The Apache-2.0-licensed llama-modchip project combines ggml and llama.cpp kernel patches, custom quantization tools, and runtime utilities for large mixture-of-experts models. Its headline benchmark runs a checkpoint named DeepSeek-V4-Flash entirely on a 16-core AMD Zen 5 CPU with 64 GiB of installed memory and no GPU offload.
| Metric | Result |
|---|---|
| Parameters | 284.33B total, 13.17B active per token |
| Quantized model size | 56.56 GiB |
| Resident memory | 57.61 to 58.90 GiB with a 256K-token KV cache |
| Prompt prefill | 43.16 to 44.36 tokens per second |
| Token generation | 9.06 to 9.95 tokens per second |
| Logit similarity | 1.00000 cosine similarity against the unpatched runtime |
A mixture-of-experts model stores many specialized feed-forward networks, then routes each token through a small subset. DeepSeek-V4-Flash therefore retains 284.33 billion parameters in memory while using 13.17 billion for each token. Prefill measures how quickly the runtime processes an existing prompt; decode measures serial generation after that prompt has been ingested.
Where the bytes go
The unmodified UD-IQ1_S build occupies 76.87 GiB, exceeding the roughly 62.5 GiB available to applications on the project’s 64 GiB Linux test system. That excess can trigger swapping, page-cache churn, or an out-of-memory termination. llama-modchip’s transcoder reduces the checkpoint to 56.56 GiB, leaving enough space for runtime state and a 256K-token KV cache. The benchmark reports zero swap use and a resident set between 57.61 and 58.90 GiB.
The transcoder assigns formats by tensor type, balancing size against sensitivity. Bits per weight, abbreviated as bpw, describes the average storage used for each quantized parameter.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
