OpenAI's Astra for Law Beats Web Search by 40% on Legal Research

OpenAI has packaged GPT-6 Astra into a specialized legal research system. Astra for Law combines the reasoning model with a legal search index, instructions for legal analysis and drafting, and settings tuned for professional use.

The legal-specific retrieval layer produced substantial gains over GPT-6 Astra with web search in OpenAI’s internal evaluation. Accuracy still remained near 54% under the benchmark’s strict scoring method, and the reported results have not received an independent audit. Vals AI, which created the benchmark, is rerunning the evaluation on its held-out test set.

GPT-6 Astra gets a legal stack

Astra for Law uses the existing GPT-6 Astra model within a system configured for United States legal research. Its search index covers case law, statutes, and regulations across more than 230 million URLs. OpenAI built the index with Free Law Project data covering more than 99.9% of published U.S. precedents.

Developers and legal teams will encounter the system under two names, depending on how they access it:

OpenAI has not announced a date, price, rate limits, or general eligibility requirements for broader API access.

Retrieval drives the gains

OpenAI evaluated the system on 200 U.S. legal research questions from the private validation split of Vals AI’s Legal Research Bench. The benchmark tests whether a system finds the relevant authorities and passages, then produces an answer containing every element required by attorney-written rubrics.

Both systems used the highest reasoning effort for the overall comparison. The increase from 38.7% to 54.0% represents a relative improvement of about 40%. Passage retrieval was compared at matching reasoning-effort levels and measured against audited target passages from the correct opinions.

Vals AI’s launch summary reports a slightly different result from OpenAI’s runs: 90.0% versus 84.1% on the weighted rubric, and 53.7% versus 39.6% on the strict all-pass measure. The published material does not explain the small difference between those figures and OpenAI’s 54.0% versus 38.7% comparison.

Leaderboard comparisons remain provisional

The public Legal Research Bench test set currently has a three-way tie at the top. Muse Spark 1.3 Max, Claude Opus 5, and Claude Fable 5.1 each score 55.29% on the strict all-pass measure. Astra for Law’s reported score of roughly 54% would place it near that group.

That comparison crosses different dataset splits: Astra for Law was measured on a private validation set, while the leaderboard uses the public test set. Its position can be assessed reliably only after Vals AI completes the held-out evaluation.

The strict metric gives credit only when an answer satisfies every required element in the attorney-authored rubric. An answer that omits one controlling authority or required conclusion fails the entire question, even if the rest is correct. This scoring explains how a system can approach 90% under weighted partial credit while passing only about 54% of questions outright.

Specialization leaves hard problems unsolved

Vals AI’s broader leaderboard analysis identifies several failure patterns that remain relevant when evaluating the specialized system:

• Reconciliation remains the hardest task. Questions involving multiple jurisdictions, conflicting precedents, or interacting regulatory regimes produce 20.7% pooled accuracy, compared with 28.4% overall. Every evaluated model performs worse on reconciliation than on other task types.
• Accuracy varies by practice area. Health and administrative or regulatory questions receive the highest scores. Family law averages 13.5% strict all-pass, while family and immigration questions rank among the hardest categories.
• Tool volume poorly predicts quality. Some systems make more than 170 tool calls per task without outperforming models that achieve similar scores with fewer than 40 calls.
• Partial credit can conceal material omissions. A weighted score above 80% may accompany an answer that misses a required authority or conclusion. Such responses can appear well supported while failing the strict rubric.

These results support a workflow in which attorneys verify authorities, quotations, jurisdiction, subsequent treatment, and the final synthesis. A 54% strict pass rate makes human review mandatory for client work, filings, and other consequential uses.

OpenAI moves deeper into legal infrastructure

The release includes 26 partner plugins from companies such as Relativity, Clio, iManage, Intapp, and DeepJudge. OpenAI also lists Zero Data Retention and governance controls developed with Latham & Watkins, plus forward-deployed engineering work with Sullivan & Cromwell, Ropes & Gray, and Cooley.

Harvey and Legora plan to build on Astra for Law through the API while continuing to offer their own legal AI products. That arrangement positions OpenAI as an infrastructure provider to companies that also compete for direct relationships with law firms.

The proprietary retrieval index gives OpenAI control over how its model searches a large body of U.S. legal material, even though the underlying case-law data comes from the Free Law Project. This approach brings the company closer to established legal research vendors whose products combine licensed or collected legal sources with search, citation tools, and editorial analysis. It also follows OpenAI’s earlier industry-specific approach for financial services and Google’s launch of Gemini Enterprise for Legal.

What developers should evaluate

Teams building legal software should test Astra for Law against their own jurisdictions, practice areas, document types, and citation requirements. The reported gains are strongest on retrieval-heavy work, particularly locating relevant cases and passages inside the correct opinions. The benchmark provides less evidence of dependable performance on cross-jurisdiction synthesis and conflicting authority.

Production evaluations should track strict task completion alongside partial-credit scores, verify every cited source, and measure tool calls, latency, and cost once API terms become available. Access is currently constrained, the benchmark results are self-reported, and the system focuses on U.S. law. Those limits should shape deployment plans until Vals AI publishes its held-out results and OpenAI provides broader API details.