---
title: "Shanghai Jiao Tong Builds AirQA to Stress-Test AI Research Agents on 14,000 Papers"
subtitle: "A new benchmark of 1,246 questions over 13,956 AI papers exposes how badly current LLM agents handle real literature-review workflows."
url: "https://alphasignal.ai/news/shanghai-jiao-tong-builds-airqa-to-stress-test-ai-research-agents-on-14-000"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-10-17T12:00:50.617Z"
updated: "2026-10-17T12:03:36.115Z"
type: "Paper"
topics:
  - "Benchmarks"
  - "Data"
subtopics:
  - "Datasets"
upvotes: 2636
source_url: "https://arxiv.org/abs/2509.16952"
comments: 0
truncated: true
---

# Shanghai Jiao Tong Builds AirQA to Stress-Test AI Research Agents on 14,000 Papers

> A new benchmark of 1,246 questions over 13,956 AI papers exposes how badly current LLM agents handle real literature-review workflows.

Research teams increasingly rely on LLM agents to search, compare, and summarize AI papers. A team from Shanghai Jiao Tong University developed AirQA to measure whether those systems answer realistic research questions accurately. Described in the [AirQA paper](https://arxiv.org/abs/2509.16952) as an ICLR 2026 benchmark, it grades individual answer fields with parameterized Python checkers and pairs the evaluation suite with a framework for generating tool-use training data.

## Paper QA needs harder targets

Earlier scientific QA benchmarks usually test comprehension of one supplied paper or retrieval of short facts from a larger corpus. Research workflows demand broader capabilities: locating a figure, comparing results across papers, extracting metadata, and returning structured outputs such as author lists or table rows.

Agent training faces a related constraint. High-quality records of multi-turn research sessions are scarce because each example requires a question, a verified answer, and a valid sequence of document searches and tool calls. AirQA addresses evaluation, while the accompanying ExTrActor framework generates trajectories for post-training.

## Inside the benchmark

AirQA contains 1,246 human-annotated examples drawn from 13,948 AI papers. Its multimodal questions cover text, tables, images, formulas, and document metadata.

| Component       | Coverage                                                      |
| --------------- | ------------------------------------------------------------- |
| Examples        | 1,246                                                         |
| Source papers   | 13,948                                                        |
| Question design | Four question types across single-paper and multi-paper tasks |

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [AirQA: A Comprehensive QA Dataset for AI Research with Instance-Level Evaluation (arXiv)](https://arxiv.org/abs/2509.16952)
- [OpenDFM/AirQA GitHub repository](https://github.com/OpenDFM/AirQA)
- [AirQA at OpenReview (ICLR 2026)](https://openreview.net/forum?id=RuUXnRyqcy)
- [NeuSym-RAG paper describing AirQA-Real statistics](https://arxiv.org/pdf/2505.19754)
