Perplexity's Open-Source WANDR Benchmark Exposes How Badly Research Agents Fail at Scale

Most research agent benchmarks ask: did the agent find the right answer? WANDR asks something harder: did the agent find all 70 qualifying companies, verify a specific fact about each one, and cite a page that actually proves it? Perplexity just open-sourced this internal benchmark, and the results reveal a sobering gap between what research agents claim to do and what they can actually deliver at scale.

The problem no one was measuring

There is a whole class of professional research work that existing benchmarks do not capture. Many useful research jobs do not end when an agent finds one answer. A market analyst may need every qualifying competitor, with the same evidence for each. A due-diligence team may need dozens of companies, then ownership, executives, financing, and regulatory status for every one of them. That creates two distinct demands that must be satisfied simultaneously:

• Wide: discover a large, open-ended set of qualifying entities
• Deep: investigate every entity far enough to support each claim with a cited, verifiable source

Combining the two changes the problem. A handful of compelling examples is not enough, and neither is a polished narrative built on incomplete research. The agent must sustain broad discovery without sacrificing factual quality from one record to the next.

What WANDR actually tests

WANDR (Wide ANd Deep Research) is an open benchmark and evaluation harness built around 500 realistic, challenging data-collection tasks for knowledge work. These are the kinds of jobs people already hand to research agents: competitive mapping, due diligence, literature review, market analysis, product comparison, talent sourcing, and more. The tasks range from dozens to thousands of independently verifiable records.

The core data structure is a qualification key hierarchy -- a tree that defines what the agent must find at each level. A task might specify:

company(n) → employee(m) → url(k)

This means: find n qualifying companies, find m qualifying employees at each, and supply k supporting URLs per employee. Every path through the tree is independently checkable. The same basic structure can represent a flat list, a nested search, a matrix, or a task with multiple evidence branches.

The median task asks for 50 members, four records per member, and 245 records overall. Taken together, the 500 tasks call for 170,495 source-backed records. Tasks are split into three difficulty tiers, and difficulty is not just about scale -- it also depends on the evidence requirements per record.

No gold answers, just live evidence checks

Most benchmarks grade against a fixed answer key. WANDR does not. A fixed answer key is a poor fit for open-ended research. It is expensive to build, quickly becomes stale, and poorly represents questions whose answers change over time. WANDR instead grades each submitted claim against the evidence the agent cites.

Every record contains an item, a URL, selected excerpts, and an answer. The grader re-fetches the page and checks whether it is usable, whether the claim is clear and in scope, whether the excerpts faithfully appear on the page, and whether both the page and excerpts support every requirement. This means certain tasks can contain time-varying facts without going stale.

The scoring uses two metrics that matter in different ways:

• Soft F1: gives partial credit to incomplete members -- useful for tracking progress
• Hard F1: requires every branch of a member's subtree to be complete and correct -- the real bar for production use

The numbers are brutal

Perplexity ran six production systems through all 500 tasks: their own Search as Code (SaC) system, Anthropic's managed agents, OpenAI's responses API, Gemini Deep Research, Exa, and Parallel. Perplexity Search as Code leads at 0.363 soft F1 and 0.133 hard F1. Anthropic is second at 0.249 and 0.072; every other system tops out at 0.121 soft F1 and 0.035 hard F1.

No system dominates both quality and efficiency. Perplexity sits in the middle of the cost range at $5.20 per task, with a 14.9-minute median solve time and 3.82 million reported tokens per task. OpenAI and Exa are faster and cheaper but score much lower. Anthropic comes closest on quality, but uses substantially more time, money, and tokens.

The effort-scaling results are equally telling. More effort improves Perplexity, Gemini, and Exa at every step; Perplexity reaches 0.447 soft F1 and 0.224 hard F1 at xhigh. Higher effort is not uniformly better, however: OpenAI peaks at high, and Parallel's hard F1 slips slightly between ultra2x and ultra4x. Cost spans more than four orders of magnitude, from $0.03 per task for Exa low to $324.83 for Gemini max.

Where agents actually fail

WANDR's per-record diagnostics reveal exactly where the breakdown happens. Four patterns stand out:

• Partial progress is the norm. The soft-to-hard conversion is harsh. Perplexity drops from 0.363 soft F1 to 0.133 hard F1, and Anthropic from 0.249 to 0.072. The best hard precision is 0.150 and the best hard recall is 0.134: even the leader earns full credit for only about one in seven members it submits.
• Scale compounds failure. From the smallest to largest target-volume bin, Perplexity's hard precision drops from 0.235 to 0.096 and its hard recall from 0.219 to 0.079. Every system degrades as the requested set grows.
• Deeper hierarchies are punishing. Going from no intermediate key to three or more, Perplexity's hard precision falls from 0.392 to 0.019 and hard recall from 0.378 to 0.017. Every added branch creates another place where a member can miss the all-or-nothing hard score.
• Discovery is the first bottleneck, evidence is the hardest. For five systems, only 3.2%--8.9% of submitted pages are unusable. The dominant losses come next: 33.6%--68.3% of submitted pages fail at least one substantive task requirement, and 57.5%--86.6% of submitted excerpts fail to support everything the record claims.

How WANDR was built

WANDR starts from de-identified patterns observed in production usage rather than synthetic prompts. The construction pipeline has four stages:

1. Seeding: mines de-identified product requests for reusable wide-research patterns
2. Authoring: runs an interleaved author-critic loop where agents sketch, stress-test, and write the task specification
3. Admission: tests feasibility by merging 10-12 authoring rollouts and audits the task-specific judge
4. Curation: selects a balanced final set across topics, scale, topology, and difficulty

This semi-automated pipeline means human effort goes into quality control rather than exhaustive answer annotation -- which is what makes 500 tasks feasible to produce.

Why Perplexity's SaC leads

Perplexity's own Search as Code system is purpose-built for exactly this task shape. Instead of one fixed pipeline behind a query, the model writes Python that composes the individual pieces of the search stack into a retrieval pipeline built for each task. A model can express retrieval, filtering, fan-out, rendering, joins, deduplication, and stopping logic as a program, while deterministic compute handles repeated operations outside the model context. On WANDR, SaC beat the next-best system by 2.5x.

What this means for the field

The benchmark exposes an assumption that needs updating: that research agent quality can be measured by whether an agent produces a good report. DRACO asks whether an agent can produce an accurate, complete, and objective long-form report; WANDR asks whether it can build a large collection and back up every member with specific evidence. These are different skills, and the field has been optimizing for the former while the latter is what professional knowledge work actually requires.

There is also a training signal angle. Rather than relying on a single sparse terminal reward, a trainer can use record- and branch-level judgments, assign partial credit for discovery and enrichment progress, and build a curriculum by raising the required counts over time. The pipeline can generate held-out sibling tasks across domains, hierarchy shapes, evidence rules, and breadth-depth settings without enumerating gold answers.

The WANDR benchmark tasks and evaluation harness are available on GitHub under an Apache-2.0 license. Running the full benchmark requires API keys for all six providers and will incur significant costs -- the README warns explicitly that the full config can be very expensive and imposes no spending cap. Start with the smoke.yaml config to validate your setup first.