Shanghai Jiao Tong Builds AirQA to Stress-Test AI Research Agents on 14,000 Papers
A new benchmark of 1,246 questions over 13,956 AI papers exposes how badly current LLM agents handle real literature-review workflows.
- AirQA benchmark covers 1,246 questions over 13,956 AI papers with function-based grading, accepted to ICLR 2026.
- 19 parameterized Python checkers enable instance-level evaluation instead of fuzzy string match or LLM-as-judge.
- Questions span 4 types and 5 element categories: text, tables, images, formulas and metadata.
- Most proprietary and open-source LLM agents underperform on the benchmark out of the box.
- ExTrActor three-agent pipeline synthesizes QA pairs and tool-use trajectories without human annotation.
- Code, data and PDFs available at github.com/OpenDFM/AirQA.
AirQA puts paper-reading agents under a field-level test
Research teams increasingly rely on LLM agents to search, compare, and summarize AI papers. A team from Shanghai Jiao Tong University developed AirQA to measure whether those systems answer realistic research questions accurately. Described in the AirQA paper as an ICLR 2026 benchmark, it grades individual answer fields with parameterized Python checkers and pairs the evaluation suite with a framework for generating tool-use training data.
Paper QA needs harder targets
Earlier scientific QA benchmarks usually test comprehension of one supplied paper or retrieval of short facts from a larger corpus. Research workflows demand broader capabilities: locating a figure, comparing results across papers, extracting metadata, and returning structured outputs such as author lists or table rows.
Agent training faces a related constraint. High-quality records of multi-turn research sessions are scarce because each example requires a question, a verified answer, and a valid sequence of document searches and tool calls. AirQA addresses evaluation, while the accompanying ExTrActor framework generates trajectories for post-training.
Inside the benchmark
AirQA contains 1,246 human-annotated examples drawn from 13,948 AI papers. Its multimodal questions cover text, tables, images, formulas, and document metadata.
| Component | Coverage |
|---|---|
| Examples | 1,246 |
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
