Skip to main content
Back

Johns Hopkins' DAR Framework Lets AI Look up Laws to Beat Legal Reasoning

DAR gives LLMs on-demand access to legal statutes as tools, pushing state-of-the-art on rule-based reasoning — but only for the strongest models

Paper, Full Access
  • DAR is a new agentic framework that gives LLMs on-demand tool access to legal statutes, rather than stuffing full rulebooks into the context window. [Paper]
  • Evaluated on DeonticBench, a 6,232-task benchmark spanning U.S. tax law, airline policies, housing law, and immigration appeals. [DeonticBench]
  • Two agentic harnesses tested: Terminus-2 (iterative retrieval loop) and Terminus-Kira (adds a verification agent for additional lookups).
  • Strong frontier models (GPT, Claude Sonnet, Kimi-K2) gain meaningfully on hard tasks; weaker models often degrade in accuracy while consuming far more tokens.
  • Best hard-subset performance on SARA Numeric reaches only 44.4% even for top models, showing deontic reasoning remains largely unsolved.
  • No public code or model checkpoint released; DeonticBench Prolog reference programs are the primary available artifact.

Legal and policy reasoning is one of the hardest tasks you can hand an LLM. Not because the language is dense, but because the rules are long, cross-referenced, and change meaning depending on which clause you read first. A new paper from Johns Hopkins University and Télécom Paris tackles this head-on with DAR: Deontic Reasoning with Agentic Harnesses, introducing an agentic framework that lets models look up statutes on demand rather than trying to hold an entire legal code in their context window.

Deontic reasoning is the formal term for applying explicit rules and obligations to specific facts to reach a conclusion. Think: computing a taxpayer's exact liability under the U.S. tax code, or determining whether an immigration appeal should succeed under USCIS administrative guidelines. It is not fuzzy or probabilistic. There is a correct answer derivable from the rules, and the model either gets it or it doesn't.

A key technical challenge for LLM-based deontic reasoning is that the relevant ruleset can be long and cross-referenced, so models may still fail to locate the rules needed for a particular reasoning step. Stuffing an entire statute into a context window is one approach, but it creates a needle-in-a-haystack problem: the model must still identify which specific sub-clause governs the current step of its reasoning, and frontier models routinely miss these lookups even when the text is right in front of them.

Across frontier LLMs and coding models, best hard-subset performance reaches only 44.4% on SARA Numeric and 46.6 macro-F1 on Housing , numbers that make clear this is not a solved problem, even for the most capable models available today.

The Benchmark: DeonticBench

DAR is evaluated on DeonticBench, a multi-domain benchmark purpose-built for rule-based reasoning. DeonticBench covers four real-world rule domains: U.S. federal taxes (SARA), airline baggage policies, U.S. state housing law, and U.S. immigration administration (USCIS-AAO). All tasks follow a unified input-output format, but the domains vary in answer type and reasoning demands, including numeric computation, logical entailment, and multi-factor legal decision-making.