Google DeepMind's MKA Benchmark Forces AI Agents to Read Blueprints Like Professionals

Google DeepMind and Artificial Analysis have introduced MKA, the Multimodal Knowledge Agent benchmark, to measure how well AI agents complete professional assignments whose decisive evidence appears in blueprints, schematics, scans, charts, and other visual files. Many existing agent benchmarks collapse into text problems after layout is removed, allowing models to work from extracted tokens alone. MKA tests visual reasoning alongside deliverable quality, execution cost, and tool use.

Inside the 200-task test

Each MKA assignment gives an agent a workspace of source files and requests a finished deliverable such as a report, spreadsheet, slide deck, or marked-up drawing. The tasks cover multi-step projects that experienced professionals routinely perform.

Domain experts built each assignment around realistic prompts, multimodal source files, and an answer rubric containing roughly 70 weighted yes-or-no checks. This structure evaluates the completed artifact instead of relying on a single reference answer.

Hard checks drive the score

Each rubric separates criteria into Climb and Regression groups. Climb criteria cover the harder requirements and determine the headline Climb weighted satisfaction rate, or Climb WSR. Regression criteria measure baseline competence, where leading models tend to post similar results.

An automated grading agent opens every submitted file, renders pages, inspects spreadsheet formulas, and compares the work with the expert rubric. Artificial Analysis uses the non-Google GPT-6 Luna model at its medium setting as the judge. In the reported validation, domain experts and the autorater agreed on 96% of grading decisions.

One sandbox, 250 turns

Every task runs through the public Stirrup harness inside a Linux sandbox with a 250-turn limit. Agents receive code execution, internet access, web search, page-fetching tools, and an image viewer when the evaluated model accepts image input. Images retain their original dimensions unless a provider imposes different limits.

Google keeps the 200 scored tasks private to reduce benchmark leakage and has released a public sample containing 10 tasks. The sample exposes the file structure, prompts, and expected deliverable formats without revealing the evaluation set.

Small score gaps, large bills

Three models lead the reported Climb WSR ranking, with only four percentage points separating first and third place. Their execution costs differ much more sharply.

The lowest reported average costs belong to several smaller or more economical configurations:

• GPT-6 Luna (Max): $0.18 per task
• Claude Haiku 5.5 (Max): $0.32 per task
• MiMo-V2.6-Pro: $0.32 per task
• GPT-6.1 Sol: about $1 per task

A single architectural-audit task illustrates how tool use produces those cost differences. Claude Fable 5.2 scored 87% after 224 turns, 203 image views, about three hours, and roughly $68 in execution costs. Gemini 4 Carbon scored 72% after 76 turns, about one hour, and approximately $8.

Specialists split the field

Results vary substantially by industry, leaving different models at the top of individual verticals:

• Claude Fable 5.2: Software & Data, Engineering, Logistics & Mapping, Business, and Finance
• Claude Opus 5.5: Education and Construction
• Gemini 4 Argon: Science
• Gemini 4 Carbon: Healthcare
• Claude Sonnet 5.5: Law & Government

Output format also changes the ranking. Word documents and plain-text files produce the weakest results, with no evaluated model exceeding 52% on either format. PDFs, spreadsheets, and presentations yield higher scores. Model selection for an agent product therefore depends on the target industry and artifact type, alongside the aggregate leaderboard position.

Where MKA helps

Because every assignment places necessary evidence inside visual material, MKA measures whether an agent can inspect that evidence and carry it into a usable professional artifact. The accompanying technical report also examines how performance changes when models lose visual access, helping isolate the contribution of multimodal perception.

Turn counts expose another deployment constraint. Evaluated models use between 30 and 130 turns per task on average, creating large differences in latency and total inference cost. For production agents, those differences can outweigh modest gaps in per-token API pricing.

What the scores leave open

Several design choices affect how the results should translate to production systems:

• The private 200-task set limits independent inspection, although it also reduces contamination risk.
• The autorater reaches 96% agreement with experts, leaving a small share of grading decisions unresolved.
• The 250-turn ceiling, sandbox, and supplied tools define a specific operating environment that may differ from a deployed agent stack.
• Cost figures depend on model settings, tool usage, provider pricing, and runtime behavior, all of which can change.

Artificial Analysis publishes the full leaderboard with example outputs. Developers can evaluate additional models under the same conditions through the public Stirrup repository.