Liquid AI's LFM2.5 Beats Bigger Rivals in Real On-Device Phone Benchmarks

Picking a language model for a phone has always been a guessing game. Benchmark leaderboards report scores on full-precision cloud builds, but what actually ships on a handset is a 4-bit quantization running on a mobile runtime with a tight memory budget and a battery to worry about. A new joint effort from Artificial Analysis and Liquid AI tries to close that gap by benchmarking the exact same quantized artifacts on real devices, including the iPhone 17 Pro and Galaxy S26 Ultra.

A benchmark built for the memory budget

A model qualifies for testing if it fits inside 8 GB of memory after quantization, including the KV cache required at 8K context. Everything is measured on the same 4-bit-or-smaller build a phone would actually load, served through llama.cpp. Intelligence gets scored on five evaluations chosen to reflect the kinds of tasks small on-device models are used for:

• BFCL for tool calling across single, multi-turn, and no-tool-fits cases
• IFBench for instruction following with verifiable output constraints
• AA-Omniscience split evenly between knowledge accuracy and hallucination resistance
• GPQA Diamond for graduate-level scientific reasoning
• MATH-500 for competition math with symbolic answer checking

The choice of evaluations matters because most headline benchmarks were built to separate frontier models, and small models cluster near the floor on them. Long agentic suites end up producing near-zero scores that fail to distinguish one 3B model from another.

The 16K context ceiling reshuffles the leaderboard

The default cap is 16K tokens per generation, which reflects how little memory is left over for KV cache on a phone. That constraint alone rearranges the rankings. Nanbeige4.2-3B and LFM2.5-2.6B share the top average evaluation scores (with a 16K context limit) at launch, both scoring 63, ahead of Ornith-1.0-9B on 62 and Qwen3.5 9B (Reasoning) on 61.

The efficiency gap between those two leaders is stark. On an iPhone 17 Pro, LFM2.5-2.6B answers a standard 1,024-token prompt in 8.0 seconds using 2.3 GB of memory, against 21.4 seconds and 4.0 GB for Nanbeige4.2-3B and more than 25 seconds and 6.9 GB for the two 9B models.

Verbose reasoning models take a real hit from the context cap. Qwen3.5 9B (Reasoning) hits the 16K context limit on 29% of its generations, which drags its scores down. With the limit raised to 64K, Ling 3.0 Tiny takes first place with a score of 66, ahead of Nanbeige4.2-3B (65), with LFM2.5-2.6B and Qwen3.5 9B (Reasoning) both on 64. A 64K window is unrealistic on a phone, however. At 55 output tokens per second on an iPhone 17 Pro, generating 64K output tokens could mean a wait of more than twenty minutes and substantial battery use.

The one-minute test flips the rankings

To simulate what a user actually tolerates, the team also caps answers at 60 seconds of on-device generation time. Capping each answer at what a model can generate in 60 seconds on the iPhone 17 Pro puts LFM2.5-8B-A1B first on 47, followed by LFM2-2.6B-Exp on 45 and Gemma 4 E4B (Non-reasoning) on 44. LFM2.5-2.6B falls to 38, and Nanbeige4.2-3B to 18: at 14 output tokens per second it gets about 850 tokens per answer, so most of its reasoning never finishes. Under this budget, verbose reasoning models never get to finish their thought, and mixture-of-experts designs that activate around a billion parameters per token pull ahead.

Where each model actually shines

The leaders have very different personalities. Nanbeige4.2-3B is the most balanced: within a point of the best on BFCL (76%), third on MATH-500 (96%) and sixth on GPQA Diamond (67%). Qwen3.5 9B (Non-reasoning) is the strongest tool caller (77% on BFCL) and the strongest scientific reasoner (79% on GPQA Diamond), while Ornith-1.0-9B recalls the most facts (15% accuracy on AA-Omniscience).

Hallucination resistance is where the spread gets uncomfortable. LFM2.5-2.6B follows instructions best of any model measured on the iPhone (59% on IFBench), clears 90% on MATH-500, and is far readier to decline than to guess: 79% non-hallucination against 33% for Nanbeige4.2-3B and 24% and 1% for the two Qwen3.5 9B variants. For an assistant that has to know when it doesn't know something, that gap is not subtle.

Try it on your own phone

The inference side runs on Pipette, Liquid AI's on-device benchmarking harness, which is open-sourced on GitHub. It runs inside a climate-controlled test facility to keep thermals from muddying the numbers, but free iOS and Android apps are also available to run the benchmarks on your own hardware.

Two practical numbers are worth keeping in mind when picking a model for shipping. Generation time spans a factor of 30, from 0.9 seconds for LFM2.5-230M to 26.7 for Falcon-H1R-7B, measured end to end on an identical 1,024-token prompt and 256-token response. Peak in-test memory at 4K context runs from 0.4 GB for the smallest models to 6.9 GB for Ornith-1.0-9B and Qwen3.5 9B, the heaviest here. On a 12 GB phone, this top end leaves limited room for the operating system and other apps.

The bigger takeaway is that mobile inference is finally being measured the way it gets used. A model's cloud leaderboard rank tells you almost nothing about how it will feel on a handset when quantization, KV cache, thermals, and verbosity all interact. For anyone shipping models smaller than 10B parameters to a device, this is the leaderboard that matches the production stack.