UC Berkeley's Agents' Last Exam Reveals AI Fails 97% of Real Professional Tasks

The AI industry has been awash in bold predictions: agents will outperform humans at nearly every job by 2026 or 2027. A team from UC Berkeley's RDI lab decided to actually test that claim. The result is Agents' Last Exam (ALE), a new benchmark that puts frontier AI agents to work on real professional tasks sourced from real experts, and scores them on the deliverables they produce. The headline finding is sobering: on the hardest tier, the average pass rate across all tested configurations is just 2.6%.

The benchmark gap that nobody was measuring

Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional domains. The researchers argue this gap is largely an evaluation problem: widely used benchmarks lack sustained performance measurement on real and economically valuable workflows. Most existing agent benchmarks either test narrow domains (SWE-bench covers software engineering, OSWorld covers desktop GUI tasks), rely on human judges, or use synthetic tasks that don't reflect how professionals actually work.

MMLU and MMLU-Pro are functionally saturated above 88% for frontier models, making score differences statistically meaningless. Humanity's Last Exam holds the best AI models to ~35% accuracy while human domain experts average ~90%. But even that benchmark only tests what a model knows, not what it can do. ALE is designed to measure the latter.

What makes ALE different

ALE is a large-scale AI agent benchmark that measures whether autonomous agents can complete real professional work, not textbook problems. Instead of scripting an agent step by step, ALE hands a frontier agent a real task on a real machine, lets it work to completion, and scores the artifacts it leaves behind against verifiable success criteria. Each task is a genuine project that a domain expert has already shipped, converted into a code-graded, fully reproducible test.

Three design principles govern which tasks get admitted:

• Representativeness: The workflow must match real professional practice using the software domain experts actually use. An architectural task should use SolidWorks or Rhino, not AutoCAD.
• Complexity: Tasks must be end-to-end deliverables that take an expert substantial time. "Apply a color filter in DaVinci Resolve" is rejected. "Move a running cheetah into a race video" (requiring tracking, rotoscoping, compositing, and color matching) is accepted.
• Verifiability: Outputs must admit deterministic checking or an unambiguous rubric. "Design an RPG game with monsters" is out. "Reproduce mota.exe using RPGMaker XP" is in, because map geometry, character attributes, and event states can be automatically compared.