Claude Fable 5 Tops Legal AI Benchmark but Fails 86% of Real Tasks

The legal AI space just got its most rigorous public scoreboard yet. Harvey LAB-AA -- Artificial Analysis's independent implementation of Harvey's Legal Agent Benchmark -- is now live, testing 28 models on 120 real-world legal tasks spanning 24 practice areas. The headline number is striking: the best model in the world, Claude Fable 5, fully completes just 14.2% of tasks. That means roughly 86% of professional legal deliverables remain incomplete, even at the frontier.

What the benchmark actually tests

Most legal AI benchmarks ask models to answer a question about a contract or analyze a clause in isolation. Harvey's LAB is built differently. Each task gives an AI agent a partner-style instruction -- the kind a senior lawyer would hand to an associate -- along with a full client matter: a set of case documents in a sandbox. The agent must read the materials, work across them, and produce a finished legal deliverable like a deal-team memo, deposition outline, or arbitration redline.

The grading is equally unforgiving. All-pass grading means a task only counts as complete if every single criterion in the rubric is satisfied -- no partial credit. As Harvey's own team puts it, a deal-team report that catches eight of ten risks isn't 80% useful; it's materially incomplete. The missing issue could change deal economics or surface as a problem after closing.

• 120 private tasks across 24 practice areas: corporate M&A, capital markets, tax, litigation, bankruptcy, IP, data privacy, and more
• Two metrics reported: all-pass rate (every criterion satisfied) and criterion pass rate (share of individual rubric items passed)
• Agent harness: Artificial Analysis's open-source Stirrup framework, with context compaction to handle long runs without failing on context limits