Harvey's Parthenon Framework Teaches Legal AI to Learn From Every Case
Parthenon introduces a self-improving legal agent framework that learns from its own failures without retraining, tested across 12,510 agent trajectories on Harvey's LAB benchmark
Takeaways
- Parthenon is a new legal agent framework that improves with experience by converting scored failures into edits to its skills, tools, and knowledge -- without retraining the model.
- A study of 12,510 agent trajectories on Harvey LAB shows frontier models improve per-criterion accuracy but stall on strict end-to-end matter completion.
- The framework uses a 6-surface MHAKTS architecture (Model, Harness, Agent roles, Knowledge, Tools, Skills) with each surface independently auditable and improvable.
- Deterministic tools handle date arithmetic, number grounding, and citation lookup -- removing hallucination risk from the parts of legal work where errors are most costly.
- Harvey LAB, the benchmark used, is open-source with 1,200+ tasks across 24 practice areas graded against 75,000 expert-written rubric criteria.
- No code or weights are released for Parthenon itself; it is a framework specification and empirical result, but the evaluation benchmark is publicly reproducible.
Legal AI has a credibility problem. Vendors make bold claims, but until recently there was no rigorous, shared way to measure whether an AI agent could actually complete a real legal matter end-to-end. A new paper from researchers Hejia Geng and Leo Liu tackles that gap head-on, combining the largest empirical study of legal agents to date with a purpose-built framework called Parthenon that gets measurably better at legal work with every case it handles, without ever touching the underlying model weights.
The Gap Between "Good at Law" and "Done with the Matter"
The legal industry is officially in the era of AI agents. Harvey AI CEO Winston Weinberg has said: "The legal industry is now well past AI as an assistant and officially in the era of legal agents." But there is a meaningful difference between an agent that scores well on individual criteria and one that can close out an entire legal matter without human intervention. The paper draws a sharp line between these two things, and the findings are sobering.
The study runs 12,510 agent trajectories through Harvey LAB, the open-source legal agent benchmark. LAB features over 1,200 tasks spanning 24 legal practice areas, underpinned by 75,000 expert-written rubric criteria. Unlike prior benchmarks that focus on short-term tasks like contract review or document comparison, LAB emphasizes long-horizon legal work, where each task mimics real-life legal workflows, requiring agents to analyze complex client matters, synthesize relevant information, and produce deliverables like risk assessments or draft memos.
The core finding: stronger models do improve per-criterion accuracy, but strict matter completion -- getting every single rubric item right in one pass -- barely budges. Frontier models are good at parts of the job. They are not yet reliable at finishing it.
Three Problems, One Framework
The paper identifies three specific obstacles blocking reliable legal agent deployment:
- No large-scale evidence on how today's best model-and-harness combinations actually perform on end-to-end legal matters
- No legal-specific architecture -- existing agent frameworks are general-purpose and not adapted to the unique demands of legal work
- No learning mechanism -- in a domain that constantly shifts with new facts, authorities, and deadlines, agents have no way to improve from their own outcomes