Artificial Analysis Ranks 11 Search APIs by How Well AI Agents Answer
A new leaderboard scores 11 search products from 7 providers on quality, cost, and speed using a shared agent harness and three benchmarks.
- Artificial Analysis launched a Search Index benchmarking 11 search products across 7 providers.
- Parallel advanced tops the board at 75, Exa auto at 74, Firecrawl and Parallel basic tied at 73.
- Every tested provider lifts quality from a model-only baseline of 33 up to 65 to 75.
- Agent runs inside Stirrup, an open-source harness with a 25-turn budget and web_search plus web_fetch tools.
- Blended scoring: DeepSearchQA F1, BrowseComp accuracy, and AA-Omniscience accuracy, equally weighted.
- Higher quality search can reduce total cost by cutting model token usage more than it adds in search fees.
Picking a search API for an AI agent has mostly been vibes work. Every provider claims freshness, ranking quality, and low latency, but there has been no head to head comparison of how those choices actually affect an agent's answers. Artificial Analysis just launched the Search Index, a leaderboard that benchmarks 11 search products across 7 providers by plugging each one into the same agent loop and measuring how well the agent performs.
The lineup covers Parallel, Exa, Firecrawl, You.com, Tavily, Keenable, and Brave. Each provider is paired with the same candidate model, GPT-5.6 Luna (medium), so the only variable in each run is what sits behind the web_search tool.
The harness that levels the playing field
The agent runs inside Stirrup, Artificial Analysis's open-source harness. The harness provides two tools: web_search and web_fetch, and the model has a 25 turn budget to gather information before it must call a finish tool to submit an answer. If the model uses all 25 turns without calling finish, no answer is submitted and the task scores zero.
Everything else is nailed down. Same reasoning effort, same temperature (0.6), up to 10 results per search, a 15 second per page fetch timeout, and contamination filtering that strips known benchmark leaks from URLs, titles, and snippets before the model ever sees them.
Three benchmarks, one blended score
The headline Search Index is the equal-weighted average of three evaluations that stress different skills:
- DeepSearchQA: 900 broad research questions with list-style answers, graded by F1 over the answer items.
- BrowseComp: a hard 200 sample subset of OpenAI's browsing benchmark that requires multi-hop navigation to find a single exact answer.
- AA-Omniscience: 600 private factual questions balanced across 6 domains, scored on accuracy.
Each provider is also compared against a model-only baseline, which is Luna answering in one shot with no tools. That baseline scores 33 on the Index, so any lift above it is attributable to search.
Who wins, and where the interesting trade-offs live
At the top of the board, Parallel Search (advanced) leads the Artificial Analysis Search Index at 75, followed by Exa Search (auto) at 74, and Firecrawl Search and Parallel Search (basic) tied at 73. Every provider tested lifts the model materially: the search-enabled runs land between 65 and 75, versus 33 for model-only.
| Provider | Search Index | Cost / 1k tasks | Time / task |
|---|---|---|---|
| Parallel Search (advanced) | 75 | $83.51 | 37.5s |
| Exa Search (auto) | 74 | $127.15 | 27.8s |
| Firecrawl Search | 73 | $75.42 | 56.9s |
| Parallel Search (basic) | 73 | $114.83 | 22.2s |
| Keenable Search (realtime) | 67 | $90.54 | 16.8s |
| Brave Search | 65 | $146.13 | 28.9s |
| Model only baseline | 33 | $2.92 | 15.9s |
A couple of counterintuitive findings jump out. Faster search calls do not automatically mean faster tasks. Parallel's turbo tier averages 0.51s per query versus 1.03s for basic, but basic scores 73 versus turbo's 67 and the two land close on total time per task because the agent needs fewer turns when results are better.
The same logic applies to cost. Parallel's advanced tier is more expensive per search than basic, but total task cost is lower ($0.084 vs $0.11) because higher quality results cut the model's token usage by over 40%. In other words, paying more for search can be a net saving on inference.
What this is actually useful for
If you are building a research agent, deep search product, or any RAG-style pipeline that reaches out to the live web, the leaderboard gives you a real comparison across three axes at once. A few practical reads:
- Best raw quality: Parallel advanced and Exa auto sit at the top of the Index.
- Best quality per dollar: Parallel Search (turbo) has the lowest measured search cost at $13.64 per 1,000 benchmark tasks, and Firecrawl offers a strong 73 Index at low blended cost.
- Best for latency-sensitive apps: Keenable Search (realtime) is the fastest, with an average time per task of 16.8s, only about a second slower than the no-search baseline.
- Slowest overall: Firecrawl, at 56.9s per task, despite being cost efficient.
The bigger contribution is methodological. By separating search provider quality from model quality and reporting cost, latency, and accuracy in the same experiment, the Search Index gives teams a defensible way to pick a retrieval backend based on measured behavior inside an agent loop rather than marketing pages.
Comments
