Google Squeezes Gemma 4 E2B Into 0.8 GB to Run on Any Device
Google's Gemma 4 E2B ships in LiteRT-LM format with mixed 2/4/8-bit quantization, hitting 91 tokens per second on phones with speculative decoding.
ProTakeaways
- Google released Gemma 4 E2B in LiteRT-LM format for Android, iOS, desktop, IoT, and Web.
- Mixed 2/4/8-bit quantization drops the text-only weight footprint to roughly 0.8 GB in memory.
- Hits 3,808 prefill and 52 decode tokens per second on Samsung S26 Ultra GPU with 0.3s TTFT.
- New speculative decoding path reaches 91.7 tokens per second on summarization tasks on-device.
- Ships KV-cache, prompt templating, and function calling via LiteRT-LM, Apache 2.0 licensed.
- Runs on Raspberry Pi 5, Jetson Orin, Qualcomm NPU, RTX 4090, and WebGPU from one artifact.
Google has pushed a deployment-ready build of Gemma 4 E2B into the LiteRT community on Hugging Face, packaged in a new .litertlm format that targets phones, laptops, single-board computers, and browsers from the same artifact. The release has already crossed a million downloads and sits near the top of trending on-device models, largely because it fits a capable instruction-tuned LLM into under a gigabyte of active memory while running at usable speeds on commodity hardware.
One artifact, six runtimes
The model card ships Gemma 4 E2B ready for deployment on Android, iOS, desktop, IoT, and web. Gemma is Google's family of lightweight open models, built from the same research that produced Gemini. This particular variant is small enough for on-device use, which gives users private access to generative AI without an internet connection.
The runtime story is where this release earns its attention. LiteRT-LM is an orchestration layer built on top of LiteRT, Google's multi-platform runtime already used across Android and edge devices. LiteRT provides hardware acceleration through XNNPack on CPU and ML Drift on GPU. LiteRT-LM adds the GenAI-specific libraries and APIs on top: KV-cache management, prompt templating, and function calling. You get the LLM plumbing without gluing together a separate inference stack per platform.
Trying it out is a one-liner once the CLI is installed:
pip install -U litert-lm
litert-lm run \
--from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
--prompt="Write me a poem"