LlamaIndex's ExtractBench Catches GPT-5 and Gemini-3 Hitting 0% on Wide Schemas

Schema-guided extraction is the quiet workhorse of enterprise AI. Agents read invoices, medical records, and shipping manifests, then pass structured JSON downstream to systems that pay, diagnose, or decide. When a model silently truncates a table or invents a value that was never on the page, the error propagates into a real transaction. ExtractBench, a new benchmark from LlamaIndex hosted on Kaggle Benchmarks, is built to catch exactly those failures.

Built for agents, not demos

Given a document and a user-defined JSON Schema, a system must return schema-valid JSON with the correct values, every record of each repeated structure, missing fields marked as null, and evidence pointing back to the source. The evaluation dataset spans 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, with tags differentiating challenge scenarios. LlamaIndex tested 14 frontier systems against it, including vision-language models, coding agents, and dedicated extraction APIs.

Where existing benchmarks fall short

Newer schema-guided benchmarks accept a schema at inference time, but each covers only a slice of the workload. None scores visual grounding, includes scanned or handwritten pages, or reports per-page cost. That gap matters when output feeds an agent acting without a human in the loop. ExtractBench targets three axes that production teams actually care about:

• Completeness on repeated structures such as line items in a long invoice or entries in a claims table.
• Null discipline: missing fields are reported as null rather than hallucinated.
• Source grounding: every extracted value can be traced to a specific location on the page.

How scoring works

All value scoring is deterministic. Each output is flattened into cells, one per scalar field and per aligned record subfield, and a cell counts as correct when it matches its expected counterpart after normalization. Precision, recall, and F1 are computed over these cells per document; slices report unweighted document means.

Grounding gets its own metric: order-insensitive value F1 for accuracy, plus word- and page-level F1 for source traceability. A field counts as grounded-correct only when its predicted bounding box overlaps a verified box on the evidence page.

A trustworthy answer key

LlamaIndex used a three-part construction pipeline: independent-system agreement for real documents, known values for synthetic lists, and human verification for forms. Where independent extractors disagreed, a human adjudicated by inspecting the document directly.

What the results reveal

The paper surfaces a clear split between system families. Commercial VLMs perform well on short documents but often truncate record lists on long ones; coding agents retain higher accuracy at much higher cost. LlamaExtract Agentic Plus ranks first on all three metrics, with accuracy comparable to coding agents at a fraction of the price.

A related study on schema breadth found a harsher failure mode. Frontier models including GPT-5/5.2, Gemini-3 Flash/Pro, and Claude 4.5 Opus/Sonnet remain unreliable on realistic schemas. Performance degrades sharply with schema breadth, reaching 0% valid output on a 369-field financial reporting schema across every tested model. Wide schemas expose failures that narrow demos never surface.

Running it yourself

The dataset, evaluation harness, and paper are all open. The GitHub repo ships a CLI you can point at your own extractor:

git clone https://github.com/run-llama/ExtractBench
cd ExtractBench
uv sync --extra runners
uv run extract-bench download --test
uv run extract-bench run llamaextract_agentic

You supply API keys only for the systems you want to evaluate, and those keys only ever call the extraction system under test. The dataset is also mirrored on Hugging Face.

What you get out of it

For teams building document workflows in supply chain, healthcare, or finance, ExtractBench provides three concrete things:

1. A way to compare VLMs, coding agents, and extraction APIs on the same schemas without writing your own harness.
2. A per-page cost signal alongside accuracy, making Pareto tradeoffs visible.
3. A grounding score showing whether an agent can defend its outputs to an auditor.

Headline value accuracy on short PDFs is a poor proxy for what happens when a 40-page contract with a 200-field schema arrives. Completeness on repeated records and honest nulls are where systems still diverge, and there is now a public leaderboard that measures both.