Google's LiteRT-LM Runs Gemma 4 at 76 Tokens/sec Without a Server
Google's open-source inference stack pushes Gemma 4 to 76 tokens/sec on-device, adds Swift and JavaScript APIs, and powers Chrome and Pixel Watch
Takeaways
- LiteRT-LM is Google's open-source C++ inference framework for running LLMs on edge devices, now at 5.4k stars
- Powers on-device AI in Chrome, ChromeOS, Pixel Watch, and the Google AI Edge Gallery app
- Multi-Token Prediction drafters deliver up to 2.2x faster Gemma 4 inference via speculative decoding
- Benchmarks show 52 tok/s on Android GPU, 56 tok/s on iOS Metal, 76 tok/s on WebGPU Mac
- Adds Swift, JavaScript, Flutter APIs plus OpenAI-compatible local server in the CLI
- Supports Gemma, Llama, Phi-4, Qwen with multi-modality, function calling, and NPU acceleration
Running a decent LLM entirely on your phone, laptop, or Raspberry Pi used to mean stitching together quantization scripts, custom kernels, and platform-specific runtimes. Google's LiteRT-LM collapses that stack into a single Apache-licensed C++ engine, and recent releases are worth a close look because the framework has quietly become the runtime powering Google's own shipping products.
LiteRT-LM is Google's production-ready, open-source inference framework for deploying large language models on edge devices. The repo has crossed 5,000 stars and sits at the C++ layer beneath the wider LiteRT (formerly TensorFlow Lite) family. It powers local AI across Chrome, ChromeOS, the Pixel Watch, and the Google AI Edge Gallery app on Android and iOS.
What's actually inside the box
A specialized orchestration layer sits on top of LiteRT, purpose-built for large language models. Google describes it as a production-proven, highly optimized runtime for running Gemma 4 on-device across Android, iOS, and the web. The interesting engineering happens above the tensor runtime. The LiteRT foundation handles limited memory, compute, and fragmented hardware by leaning on advanced quantization schemes along with accelerated XNNPACK and MLDrift kernels. Above that, optimized pipelines minimize costly CPU-GPU data transfers, support multi-token prediction, and provide session management for stateful workloads.
The headline capabilities:
- Cross-platform reach: Android, iOS, Web, Desktop, and IoT such as Raspberry Pi.