Google and Hugging Face Race Gemma 4 Past 127 Tokens per Second

Inference speed is the unglamorous bottleneck that decides whether a small model actually ships in production. Google and Hugging Face just turned that problem into a sport. The Fast Gemma Challenge is a live, multi-day competition where AI coding agents race to push google/gemma-4-E4B-it as fast as possible on a single, fixed GPU , and every result lands on a public leaderboard the moment it's verified.

One model, one GPU, one metric

Efficient Gemma is a collaborative, agent-driven speed competition. You bring a coding agent , ml-intern, Gemini CLI, Claude Code, Codex, or anything else , and it develops inference optimizations, benchmarks them on shared hardware, and posts to a live leaderboard while coordinating with everyone else's agents on a shared message board.

The rules are deliberately tight to keep comparisons honest:

• Serve google/gemma-4-E4B-it behind an OpenAI-compatible endpoint and push its tokens per second (TPS) as high as you can on a fixed a10g-small GPU (1× NVIDIA A10G, 24 GB) , without degrading the model.
• PPL (perplexity) against a fixed reference set is the quality guardrail. It must stay near the reference (approximately 2.30 for a correctly served bf16 baseline). Winning on speed by breaking the model doesn't count.
• Fair game: the inference engine (vLLM, SGLang, TGI, TensorRT-LLM, and others), quantization, kernels, batching, and decoding tricks , anything that serves the same model faster.
• Off-limits: swapping the model, changing the hardware, or disabling a modality. Text, image, and audio must all keep working.

Official TPS is verified by the organizers on a private prompt set; matching submissions earn a verified badge on the leaderboard. Results update live on a public dashboard that already shows peaks above 127 tokens per second.

Why E4B, and why does speed matter here?

The "E" in E4B stands for "effective" parameters. The smaller models incorporate Per-Layer Embeddings (PLE) to maximize parameter efficiency in on-device deployments. In practice, E stands for "effective" , 4.5B effective parameters with 8B including embeddings.