Liquid AI's Pipette Exposes What Server Benchmarks Hide About Phone AI

Model cards keep telling you what a language model can do on an H100. They rarely tell you what happens when you shove a Q4 quant of that same model onto an iPhone with a 4,000-token prompt. Liquid AI is trying to close that gap with Pipette, an open-source benchmarking suite built with Artificial Analysis that measures foundation models on the actual devices people ship them on.

The premise is straightforward: on-device behavior is a property of the deployed system, not the model in isolation. Instead of scoring a checkpoint in a vacuum, every measurement is tied to a full deployment configuration of model + quantization + runtime + device. Change any one of those and you get a different row in the database.

What Shipped

The launch is a full stack rather than a single dashboard. Here is what is available today:

• A public dataset of lab-verified results covering more than 1,000 configurations across models, quantizations, runtimes, devices, and context lengths. The current data spans 30+ models, multiple quantization formats, llama.cpp builds for macOS, iOS, Windows, and Android, and context lengths from 256 to 8,192 tokens. Initial published results cover MacBook Pro with M5 Max, iPhone 17 Pro, and Galaxy S26 Ultra, with AMD Ryzen AI Max+ 395 and Radeon 8060S results coming soon.
• Open-source benchmark clients for macOS, Windows, iOS, and Android, including native iOS and Android apps for executing performance benchmarks directly on target devices.
• An interactive leaderboard plus a companion view on Artificial Analysis.
• Apache 2.0-licensed infrastructure for operating the complete pipeline, including pipette-mgmt, pipette-clients, and pipette-scores.

Why Server Benchmarks Lie About Phones

Model releases commonly include capability scores produced under server-class, full-precision conditions. The moment you quantize to Q4 and drop it on a Snapdragon, most of that signal degrades in ways nobody publishes. Pipette pairs quantized performance runs on real hardware with quality evaluations on IFBench, GPQA Diamond, and MATH-500, with FP16 or BF16 results providing a reference where available.

The examples Liquid highlights make the case better than any pitch. At Q4_K_M on Galaxy S26 Ultra, Granite-4.0-H-350M retains 78.4% of its decode throughput from 256 to 4,096 input tokens, while Granite-4.0-350M retains only 33.8%. Two models with identical parameter counts, identical quantization, identical hardware, wildly different context scaling.

Sparsity flips assumptions too. At 2,048 input tokens on Galaxy S26 Ultra, LFM2.5-8B-A1B decodes 2.4x faster than Qwen3.5-4B and 2.6x faster than Ministral-3B-Instruct-2512. Despite activating only 1.5B of its 8.5B parameters per token, it still peaks at 5.29 GiB because all expert weights contribute to the memory footprint. You get small-model speed with big-model RAM pressure, exactly the kind of tradeoff a leaderboard number would hide.

How The Measurements Work

The reproducibility protocol is unusually strict for this space. Timing and memory runs use fixed token shapes, greedy decoding, a discarded warm-up, five measured repetitions, and readiness gating. That readiness gate matters: before each timed repetition, a platform-specific check verifies acceptable thermal and load conditions, so a phone that has been throttling for ten minutes does not get to log a result.

Three components split the pipeline cleanly. pipette-mgmt serves versioned benchmark definitions, pipette-clients runs inference on target devices, and pipette-scores grades completions without seeing their generation provenance, keeping scoring model-blind.

Where It Falls Short

The team is honest about what Pipette does not yet do. NPU support depends on model-specific kernel and operator coverage. In this initial release, no NPU path supports enough of the published model set to provide consistent comparisons across model classes, so NPU results are omitted. Android GPU backends are similarly excluded because none of the stable GPU backends tested on Android consistently outperformed the selected CPU path across the full model set.

Cross-device comparisons are also off-limits for now. Android and iOS runs differ in flash-attention support, thread counts, accelerator use, and execution environment. Android runs use a CPU-based CLI path, while iOS runs inside the app using Metal. Current results are only reliable for comparing configurations within the same device.

Who Should Actually Care

If you are shipping a small language model into a mobile app, a laptop assistant, or an embedded product, this is the first public dataset that lets you pick a quant with your eyes open. You can inspect the runtime, quantization, and device you plan to ship, and if that configuration is missing, run the clients yourself and contribute back. Every accepted submission expands coverage, which is the mechanism Liquid is betting on to keep the dataset current as new models and phones drop.

The documentation is thorough enough to be replicable, and the iOS and Android apps mean you do not need a lab rig to contribute a data point. For a category that has been dominated by vibes and vendor slides, a versioned, model-blind, thermally-gated benchmark suite is a meaningful shift.