Perplexity's Open-Source WANDR Benchmark Exposes How Badly Research Agents Fail at Scale
Perplexity open-sources WANDR, a 500-task benchmark exposing how badly research agents fail at large-scale, evidence-backed data collection
- WANDR open-sourced: Perplexity releases a 500-task benchmark testing agents on large-scale, evidence-backed data collection (GitHub)
- Scores are low across the board: The best system (Perplexity SaC) hits only 0.363 soft F1 and 0.133 hard F1 -- wide-and-deep research is far from solved
- Evidence extraction is the hardest part: 57-86% of submitted excerpts fail to fully support their claims across all tested systems
- Scale kills performance: Hard F1 drops sharply as task size grows; deeper hierarchies compound the failure at every added branch
- No gold answers needed: WANDR re-fetches every cited page live, enabling time-varying facts and avoiding stale answer keys
- Doubles as training data factory: Per-record verdicts provide dense reward signals for RL training without requiring exhaustive manual annotation
Most research agent benchmarks ask: did the agent find the right answer? WANDR asks something harder: did the agent find all 70 qualifying companies, verify a specific fact about each one, and cite a page that actually proves it? Perplexity just open-sourced this internal benchmark, and the results reveal a sobering gap between what research agents claim to do and what they can actually deliver at scale.
The problem no one was measuring
There is a whole class of professional research work that existing benchmarks do not capture. Many useful research jobs do not end when an agent finds one answer. A market analyst may need every qualifying competitor, with the same evidence for each. A due-diligence team may need dozens of companies, then ownership, executives, financing, and regulatory status for every one of them. That creates two distinct demands that must be satisfied simultaneously:
- Wide: discover a large, open-ended set of qualifying entities
- Deep: investigate every entity far enough to support each claim with a cited, verifiable source
Combining the two changes the problem. A handful of compelling examples is not enough, and neither is a polished narrative built on incomplete research. The agent must sustain broad discovery without sacrificing factual quality from one record to the next.
What WANDR actually tests
WANDR (Wide ANd Deep Research) is an open benchmark and evaluation harness built around 500 realistic, challenging data-collection tasks for knowledge work. These are the kinds of jobs people already hand to research agents: competitive mapping, due diligence, literature review, market analysis, product comparison, talent sourcing, and more. The tasks range from dozens to thousands of independently verifiable records.
The core data structure is a qualification key hierarchy -- a tree that defines what the agent must find at each level. A task might specify:
company(n) → employee(m) → url(k)This means: find n qualifying companies, find m qualifying employees at each, and supply k supporting URLs per employee. Every path through the tree is independently checkable.
The same basic structure can represent a flat list, a nested search, a matrix, or a task with multiple evidence branches.