Google packages Gemma 4 12B for offline laptop inference

Google has published a laptop-ready build of Gemma 4 12B in the .litertlm format. The instruction-tuned model accepts text, image, and audio inputs, generates text, and can run locally on a 16GB M4 MacBook Air once its files are downloaded.

Google converted the checkpoint for LiteRT-LM, an orchestration layer built on LiteRT, formerly TensorFlow Lite. LiteRT handles model execution and hardware acceleration, while LiteRT-LM manages prompt formatting, multimodal preprocessing, KV caches, token generation, and tool calls.

Inside the 6.9GB bundle

The 6,883MB package supports text, vision, audio, and Multi-Token Prediction, or MTP. The desktop build targets macOS, Linux, and Windows and requires LiteRT-LM v0.17 or later.

MTP generates several candidate future tokens during one model pass. The runtime verifies those candidates before accepting them, reducing sequential decoding work when the predictions match and lowering latency on supported hardware.

The repository metadata lists the packaged release under Apache-2.0. Teams planning to redistribute it should also review the model card for any additional terms covering the underlying Gemma weights.

From download to first token

Installing LiteRT-LM v0.17 or later and pointing its CLI at the Hugging Face repository starts an interactive run:

pip install -U litert-lm
litert-lm run \
  --from-huggingface-repo=litert-community/gemma-4-12B-it-litert-lm \
  --prompt="Write me a poem"

The first run requires a network connection to install the runtime and download roughly 6.9GB of model data. Subsequent inference can remain offline, provided the surrounding application does not send prompts, telemetry, or generated output to external services.

The AI Edge Gallery app can also load the model on macOS. Its configurable image-token budget lets developers trade image detail and context usage against memory consumption. Application APIs are available for C++, Python, Kotlin, Swift, JavaScript, and Flutter.

Hardware sets the pace

Google measured performance with a 1,024-token prompt, 256 generated tokens, and a 4,096-token context window. The results reflect complete platform stacks, including the operating system, drivers, backend, and hardware, so direct GPU comparisons require caution.

Apple silicon uses unified memory shared by the CPU, GPU, operating system, and applications. A reported footprint near 7.9GB therefore consumes roughly half of a 16GB MacBook Air’s total memory before accounting for the operating system and other processes.

The MacBook Air’s 9.1-second first-token result largely reflects the time required to process the 1,024-token benchmark prompt at 114 tokens per second. Shorter prompts should begin generating sooner. These figures cover one prompt shape and do not establish model download or cold-start loading times.

Context length meets memory limits

LiteRT-LM can expose Gemma 4 12B’s context window up to 128K tokens when sufficient memory is available. Each additional token expands the KV cache, which stores the model’s attention state, so practical limits depend on device memory, image-token budgets, prompt length, and requested output length.

The model card recommends reducing max_num_tokens when memory allocation fails. Google’s laptop benchmarks use a 4,096-token window, making the published performance figures a better guide for constrained machines than the 128K maximum.

Browser deployment uses a separately optimized artifact because browser runtimes impose different memory and buffer limits. The current web build accepts text only and was benchmarked in Chrome with a 1,280-token context.

Deployment fits and constraints

• Offline multimodal assistants: Text, images, and audio can remain on the device after installation, subject to the host application’s own network behavior.
• Tool-using applications: LiteRT-LM supports function calling with constrained decoding, which can improve adherence to required argument schemas.
• Discrete NVIDIA GPUs: The published results reach 50 to 69 generated tokens per second, with faster prompt processing on the tested Linux RTX 4090 system.
• Entry-level Apple silicon: The tested 16GB MacBook Air produced 15 tokens per second and required 9.1 seconds to process the benchmark prompt before generation.
• Long-document workloads: KV-cache growth makes the 128K maximum impractical on many memory-constrained laptops.
• Browser applications: The separate web artifact currently omits image and audio input.

A Google-backed route to local deployment

Local 12B multimodal inference is already available through projects such as llama.cpp and MLX. LiteRT-LM gives teams using Google’s edge stack a maintained deployment path with model loading, cache management, multimodal processing, MTP, and constrained tool calls under one API layer.

LiteRT supplies hardware backends such as XNNPack for CPU execution and ML Drift for GPU acceleration, while LiteRT-LM exposes interfaces for desktop, mobile, and cross-platform applications. Browser deployment still requires its own optimized model, and throughput varies substantially across operating systems and accelerators.

Developers building local assistants, document tools, or offline mobile features can prototype through the CLI and retain the same runtime family when moving into application code. The concrete trade-offs are an approximately 8GB runtime memory footprint, hardware-dependent latency, and context capacity governed by available memory.