Skip to main content
Back

Claude Fable 5 Tops Legal AI Benchmark but Fails 86% of Real Tasks

Artificial Analysis launches Harvey LAB-AA, a rigorous legal agent benchmark where Claude Fable 5 leads at just 14.2% -- revealing how far AI still is from autonomous legal work

  • Harvey LAB-AA launches: Artificial Analysis releases an independent implementation of Harvey's Legal Agent Benchmark, testing 28 models on 120 real-world legal tasks across 24 practice areas.
  • Claude Fable 5 leads at 14.2% all-pass -- nearly double the next tier -- but still fails to fully complete 86% of professional legal deliverables.
  • All-pass grading is strict: a task only counts as complete if every single rubric criterion is satisfied; 13 of 28 models score 0%.
  • GLM-5.2 is the open-weight standout: ties Claude Opus 4.8 at 7.5% all-pass for ~6x lower cost (~$1.30 vs. ~$8.20 per task) and 3x faster.
  • Cost spans 950x: from ~$19/task (Claude Fable 5) to ~$0.02/task (Gemini 3.1 Flash-Lite), with massive variation in what you get for the price.
  • Benchmark and harness are open-source: explore results at Artificial Analysis and the task set at harveyai/harvey-labs.

The legal AI space just got its most rigorous public scoreboard yet. Harvey LAB-AA -- Artificial Analysis's independent implementation of Harvey's Legal Agent Benchmark -- is now live, testing 28 models on 120 real-world legal tasks spanning 24 practice areas. The headline number is striking: the best model in the world, Claude Fable 5, fully completes just 14.2% of tasks. That means roughly 86% of professional legal deliverables remain incomplete, even at the frontier.

What the benchmark actually tests

Most legal AI benchmarks ask models to answer a question about a contract or analyze a clause in isolation. Harvey's LAB is built differently. Each task gives an AI agent a partner-style instruction -- the kind a senior lawyer would hand to an associate -- along with a full client matter: a set of case documents in a sandbox. The agent must read the materials, work across them, and produce a finished legal deliverable like a deal-team memo, deposition outline, or arbitration redline.

The grading is equally unforgiving. All-pass grading means a task only counts as complete if every single criterion in the rubric is satisfied -- no partial credit. As Harvey's own team puts it, a deal-team report that catches eight of ten risks isn't 80% useful; it's materially incomplete. The missing issue could change deal economics or surface as a problem after closing.

  • 120 private tasks across 24 practice areas: corporate M&A, capital markets, tax, litigation, bankruptcy, IP, data privacy, and more
  • Two metrics reported: all-pass rate (every criterion satisfied) and criterion pass rate (share of individual rubric items passed)
  • Agent harness: Artificial Analysis's open-source Stirrup framework, with context compaction to handle long runs without failing on context limits
  • Judge: A single Gemini 3.1 Pro model, validated against a frontier panel for calibration

The leaderboard, and what it reveals

Claude Fable 5 (with an Opus 4.8 fallback on 1 task) leads at 14.2% all-pass -- nearly double the next tier. Claude Opus 4.8 and GLM-5.2 from Z AI tie for second at 7.5%. MiniMax-M3 follows at 6.7%, Claude Sonnet 5 at 5.0%, and GPT-5.5 at 4.2%. Of the 28 models evaluated, 13 fully pass zero tasks.

ModelAll-Pass RateCriteria Pass RateCost per TaskAvg. Time per Task
Claude Fable 5 (w/ fallback)14.2%93.6%~$18.9~16.9 min
Claude Opus 4.8 (max)7.5%91.1%~$8.2~18.5 min
GLM-5.2 (max)7.5%91.0%~$1.3~5.0 min
MiniMax-M36.7%------
Claude Sonnet 55.0%90.1%~$11.8~22.8 min
GPT-5.5 (xhigh)4.2%------
Gemini 3.1 Flash-Lite~0%31.1%~$0.02--

The open-weight surprise

The most interesting cost-efficiency story belongs to GLM-5.2. The open-weight model from Z AI ties Claude Opus 4.8 on both all-pass rate (7.5%) and criteria pass rate (91.0%), while costing roughly 6x less per task (~$1.30 vs. ~$8.20). It also completes tasks in about 5 minutes on average, compared to Opus 4.8's 18.5 minutes. For teams that need to run legal agent workflows at scale, that gap is significant.

The cost spread across the full leaderboard is extreme: ~950x separates the most expensive model (Claude Fable 5 at ~$19/task) from the cheapest (Gemini 3.1 Flash-Lite at ~$0.02/task). The cheap end passes almost no tasks in full, but passes 31% of individual criteria -- which may be useful for triage or pre-screening workflows.

More tokens, more turns -- but diminishing returns

A clear pattern emerges when you look at how the leading models work: they grind. Claude Fable 5 averages ~64 turns per task and generates ~117k output tokens to reach its 14.2% score. Claude Sonnet 5 runs the longest loops of any model -- ~161 turns and ~179k output tokens per task -- yet only reaches 5.0% all-pass. More compute doesn't automatically translate to better legal work.

This points to a real limitation: models are good at satisfying most individual rubric criteria (the top four models all exceed 90% criterion pass rate), but they consistently miss at least one requirement per task. The failure mode isn't ignorance -- it's incompleteness. A model might draft a thorough M&A memo but miss one severity rating, one consent timeline, or one cross-document inconsistency that a partner would catch immediately.

How it's built, and where it differs from Harvey's original

Harvey LAB-AA is Artificial Analysis's independent reimplementation, not a direct port. Several deliberate choices distinguish it:

  • No custom document-generation tools: Harvey's original includes specialized scripts for producing .docx and .pptx files. LAB-AA strips these out and uses a simple code execution tool, testing raw model capability rather than tool-assisted output
  • Strict filename matching: Deliverables must match the exact filename specified, rather than fuzzy LLM-based matching
  • Context compaction: When models approach context limits, the harness compacts history rather than failing the task outright -- a more realistic reflection of production agent behavior
  • Single judge: Grading uses one Gemini 3.1 Pro model, tested for calibration against a frontier panel

Why this benchmark matters now

Legal AI has been evaluated almost exclusively on short-horizon tasks: answer a question about a contract, compare two clauses, classify a provision. Harvey built LAB specifically to fill the gap, providing transparency into how agents perform on real legal tasks and the trade-offs associated with key agent design decisions like model selection, harness optimization, and post-training. The benchmark mirrors how work is actually assigned at a law firm -- a loose partner instruction, a pile of documents, and a requirement to produce something a partner would sign off on.

The pace of development at the model layer is increasing, with major labs pushing the frontier of general intelligence on a monthly basis. The cost of operating at that frontier rises with it, as the best-scoring models are increasingly token-hungry and reliant on time-consuming, multi-step reasoning. LAB-AA makes those trade-offs legible in a way that matters for anyone deciding which model to deploy in a legal workflow.

The 10% threshold matters because it is where the failure modes shift -- below 10%, models can handle isolated tasks with human review of every output; above 10%, the class of tasks where the model can operate more autonomously begins to expand. Claude Fable 5 is the first model to cross that line on this benchmark. The gap between it and every other model is currently large -- but the benchmark is public, the harness is open-source, and the research community is already running experiments. One early result from Fireworks shows a hybrid harness pairing an open-source GLM worker with Claude Opus 4.7 as an advisor reaching 18/100 all-pass at $368, while fine-tuning Kimi K2.6 on LAB trajectories reaches 15/100 at $84. The benchmark is already doing what it was designed to do: accelerating research.