CMU's AgentWebBench Reveals Where AI Agents Break When Searching the Web
A new benchmark tests how well LLM agents coordinate with website-specific agents to retrieve information, revealing that decentralized access still trails centralized search but the gap closes with scale.
ProTakeaways
- AgentWebBench (ICML 2026) benchmarks multi-agent coordination in the Agentic Web across 100 websites and 18.4M documents. Paper
- Four tasks: web search, recommendation, QA, and deep research, with 354, 281, 53, and 331 samples respectively.
- Multi-agent coordination generally trails centralized retrieval, but the gap closes with model scale.
- On QA, Gemini-3 Multi-Agent beats Classical (67.92% vs 60.38% accuracy) via iterative evidence gathering.
- Decentralized access concentrates traffic on Wikipedia, ScienceDirect, and other authoritative sources.
- Code, data, and hosted API released at github.com/cxcscmu/AgentWebBench.
The web is quietly bifurcating. On one side, users increasingly delegate browsing, research, and shopping to LLM-driven assistants. On the other side, content providers are locking down raw data and exposing it only through APIs and agent-facing interfaces, often mediated by protocols like MCP and A2A. AgentWebBench, an ICML 2026 benchmark from Carnegie Mellon, is the first serious attempt to measure what happens when your agent has to negotiate with someone else's agent to get information out.
The paper formalizes this emerging setup as the Agentic Web: a decentralized ecosystem where a user agent selects websites, queries their content agents through structured interfaces, and synthesizes evidence into answers. In this paradigm, user agents coordinate with multiple autonomous content agents across information-seeking tasks. Crucially, the user agent has no direct access to the underlying corpus, which mirrors the real-world trend of publishers restricting scraping while offering agent-mediated endpoints.

Why single-agent benchmarks miss the point
Existing web benchmarks like WebArena and Mind2Web mostly test a single agent clicking through a browser. That framing misses the coordination problem entirely, focusing on solo performance rather than the negotiation demands of the emerging Agentic Web. When information access requires working with multiple autonomous agents that each control their own retrieval infrastructure, the failure modes are different, and so are the metrics that matter.
How the benchmark is built
AgentWebBench uses the English subset of ClueWeb22-B and picks the 100 websites with the largest document counts, yielding a decentralized set of sources that collectively contain 18,427,770 documents. Each site is served by its own content agent that maintains an independent dense retrieval index built with MiniCPM-Embedding-Light. The sites span the diversity you'd expect: Amazon, Wikipedia, Stack Overflow, GitHub, ScienceDirect, PubMed, Yelp, Reddit, and 92 others.