Notre Dame's PolicyMoE Beats GPT-4o on Government Policy by 51%

Public policy is one of the messier places to deploy an LLM. A wrong answer about tax rules, funding formulas, or environmental regulations can quietly translate into under-allocation of resources or non-compliant decisions. A team from the University of Notre Dame has built the first large-scale benchmark that measures how well current models understand policy text, and they released a specialized Mixture-of-Experts model to close some of the gaps it exposes.

The work, accepted at ACL 2026 Findings, introduces two artifacts: PolicyBench, a 21K-case cross-system benchmark covering US federal and mainland Chinese policy, and PolicyMoE, a domain-specialized MoE built on Qwen2.5-7B-Instruct. Dataset and code are on GitHub.

Why general-purpose LLMs stumble on policy

The field's interdisciplinary nature, contextual dependence, and linguistic complexity are well recognized, but the central obstacles to policy-aware AI can be framed as a three-tiered problem: the absence of rigorous benchmarks to assess policy comprehension, the difficulty of diagnosing where models succeed and fail across skills, and the need for specialized architectures. Aggregate leaderboard scores tend to hide exactly the failure modes that matter in governance settings, like misreading institutional logic or hallucinating procedural steps.

The authors chose the US federal government and mainland China as their two policy environments. Across experiments, model performance improves steadily from memorization to application tasks, with LLMs showing particular strength in structured reasoning such as numerical calculation and scenario-based decision-making, while still struggling with abstract or ambiguous policy contexts and Chinese policy texts.