Google's Android Bench Crowns Claude Fable 5 Top AI Coding Assistant at 84.5

Google's Android Bench just got its biggest overhaul since launch. The leaderboard, which measures how well LLMs handle real Android development tasks, has added 8 new models, swapped out its evaluation engine for the Harbor Framework, and opened the door for community-submitted tasks. If you're picking an AI coding assistant for Android work, this is the most comprehensive signal you have.

The scores that matter

Claude Fable 5 sits at the top of the leaderboard with a score of 84.5, followed by GPT 5.5 with 80.2, and Claude Sonnet 5 in third with a score of 76.2. That's a meaningful jump from the previous generation of models. For context, the original release saw models successfully complete between 16% and 72% of tasks -- the ceiling has moved significantly.

On the open-weight side, GLM 5.2 leads with 72.2, followed by Kimi K2.7 Code with a score of 70.4. That's a strong result for open models, putting them within striking distance of last generation's proprietary leaders.

The 8 new additions to the board are:

• Claude Fable 5 -- new #1 overall at 84.5
• GPT 5.5 -- #2 overall at 80.2
• Claude Sonnet 5 -- #3 overall at 76.2
• Claude Opus 4.8
• GLM 5.2 -- top open-weight at 72.2
• Kimi K2.7 Code -- #2 open-weight at 70.4
• MiniMax M3
• Qwen 3.7 Plus and Qwen 3.7 Max

Why a new evaluation engine changes everything

The bigger story here isn't the scores -- it's the methodology shift. Android Bench previously ran on mini-swe-agent v1, a general-purpose benchmarking agent adapted for Android. As part of the July release, Google has adopted the Harbor Framework, which includes an updated version of the benchmarking agent used to evaluate models.

Harbor is a framework designed for evaluating and optimizing AI agents and language models within sandboxed container environments, covering the entire lifecycle of agent evaluation runs -- spanning isolated environment provisioning, agent installation and execution, result verification through test scripts, recording agent interaction trajectories, and exporting structured training datasets. In plain terms: each model gets dropped into a clean, isolated container, given a real Android task, and its output is verified by actual tests -- not a judge model guessing whether the code looks right.

The team says Harbor creates sandbox environments to test and evaluate agents, and the upgraded methodology makes it easier for anyone to run the benchmark, evaluate their preferred setup, or share results. Because the framework is standardized, results are now reproducible and comparable across different teams running the same benchmark independently.

The switch does mean a minor scoring shift across the board. Google re-ran the benchmark on all models to establish an updated baseline, and historical scores remain viewable in the archive.

What Android Bench actually tests

Android developers face specific challenges that aren't covered by existing benchmarks, so Google created one that focuses on a north star of high-quality Android development. General coding benchmarks often fail to capture the platform-specific dependencies and nuances of mobile development, and Android Bench addresses this by curating a task set sourced directly from real-world, public GitHub Android repositories.

The task types cover exactly the things that trip up AI assistants in practice:

• Resolving breaking changes across Android releases, domain-specific tasks such as networking on Wear OS devices, and migrating code to the latest version of Jetpack Compose.

Code generation is evaluated based on functionality, not methodology -- the framework automatically verifies the LLM's proposed fixes using standard Android engineering practices: isolated unit tests and emulator-based instrumentation tests. There's no LLM-as-judge involved; the code either passes the tests or it doesn't.

To keep results honest, the benchmark employs manual reviews of agent reasoning paths and uses canary strings to prevent AI web crawlers from ingesting the test dataset -- a direct countermeasure against models memorizing benchmark answers during training.

The community layer

The most forward-looking part of this update is the community contribution system. Android Bench is being opened to the global developer community, allowing anyone to submit their own custom Android dev tasks and share benchmark evaluations.

Concretely, you can now do two things via GitHub:

1. Submit your own Android development tasks -- scenarios from your actual work that you want models tested against.
2. Run the benchmark yourself against the official dataset and share your results on Harbor Hub.

This matters because the benchmark's validity depends on task diversity. A leaderboard built entirely by one team at Google will inevitably reflect Google's definition of "good Android development." Opening submissions to the broader community is how you get tasks that reflect real-world diversity -- obscure APIs, legacy codebases, niche Wear OS scenarios -- rather than a curated ideal.

What this means for your model selection

The practical upshot is straightforward. Claude Fable 5 leads overall at 84.5, while GLM 5.2 leads among open-weight models at 72.2 -- so if you're cost-sensitive or need to run locally, GLM 5.2 is now the open-weight benchmark to beat for Android work. The leaderboard also includes cost and efficiency dimensions alongside raw scores, so you can weigh performance against token cost for your specific use case.

The Harbor migration also means you can now run the benchmark yourself with a few CLI commands, verify results independently, and even test models that aren't on the official leaderboard yet. For teams building Android-specific AI tooling or evaluating which model to integrate into their CI pipeline, that reproducibility is the real unlock.