Google's gemma.cpp Runs Gemma 3 on CPU With No GPU Required
Google's gemma.cpp brings Gemma 2 and 3 inference to pure C++ with no GPU needed, targeting researchers who want to hack on model internals
PRO- gemma.cpp is Google's open-source C++ inference engine for Gemma 2, Gemma 3, and PaliGemma 2, running entirely on CPU with no GPU required.
- The core implementation is ~2K lines of C++, designed to be readable and hackable for research into quantization, decoding, and model internals.
- Uses Google's Highway SIMD library for portable, runtime-dispatched CPU acceleration across AVX2, AVX-512, NEON, and more.
- Supports mixed-precision inference (fp8, bf16, fp32, NUQ 4-bit), a backward pass with Adam optimizer, and batched streaming inference.
- Free and available now on GitHub under Apache 2.0; weights available on Kaggle and Hugging Face.
- Not recommended for production GPU deployments; Google points those use cases to JAX, PyTorch, and Transformers instead.
gemma.cpp is Google's lightweight, standalone C++ inference engine for running Gemma models directly on CPU, with no GPU required. It supports Gemma 2, Gemma 3, and PaliGemma 2, and is built for a specific kind of user: someone who wants to poke around inside the model, not just call it as a black box. Think of it as the spiritual cousin of llama.c and ggml, but purpose-built for Google's own model family.
The gap it fills
gemma.cpp targets experimentation and research use cases, and is intended to be straightforward to embed in other projects with minimal dependencies and easily modifiable, with a small ~2K LoC core implementation (along with ~4K LoC of supporting utilities). That's the key pitch. Most C++ inference runtimes are production-hardened and opaque by design. Python ML frameworks, on the other hand, abstract away the low-level compute through compilation. gemma.cpp sits in the middle: readable, hackable, and fast enough to be genuinely useful.
The author, Austin Huang, put it plainly in the Hacker News thread at launch: the weights should be the same across formats, but differences can arise due to quantization and subtle implementation differences, which has been a pain point in the ML ecosystem for a while. The project is explicitly not trying to replace llama.cpp or vLLM for production. It's for the people who want to add sparsity support, custom quantization, or a novel decoding strategy and see it working with a real model in minimal code.
What's under the hood
The project uses the Google Highway Library to take advantage of portable SIMD for CPU inference. SIMD (Single Instruction, Multiple Data) is a CPU technique that processes multiple data points in a single instruction cycle, which is how you squeeze real throughput out of a CPU for matrix math. Highway is notable because it picks the best available instruction set (AVX2, AVX-512, NEON, etc.) at runtime, so the same binary runs efficiently on any machine without recompilation.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
