Perplexity's Q2D-Web Benchmark Tests AI Search on 190M Real Web Documents

Perplexity has released Q2D-Web, a private benchmark and public leaderboard designed to test how embedding models handle the messy, agent-reformulated queries that drive modern retrieval-augmented generation. It covers 190 million web documents and 69,721 agent-reformulated queries in ten languages, sampled over nine months of PII-free production search traffic. The full technical write-up is on arXiv, and the live rankings sit on a Hugging Face Space.

Where existing retrieval benchmarks fall short

Most retrieval evaluations scale along only one axis: either a big corpus with few judged queries, or many queries against a tiny corpus with shallow labels. Small corpora strip out the hard negatives that make retrieval difficult, and sparse labels punish good retrievers for finding relevant documents that were never annotated. On MS MARCO, 70% of manually inspected top-retrieved but unlabeled passages were in fact relevant, meaning models were being penalized for correct answers.

Q2D-Web also targets a mismatch specific to agentic systems. Benchmarks typically use human-written queries, whereas RAG agents rewrite user requests into their own reformulated queries with different wording, structure, and specificity.

What is actually in the benchmark

Q2D-Web contains 190 million web documents, 69,721 agent-reformulated queries, and an average of 99.6 positive relevance judgments per query in its combined judgment set. For comparison, MS MARCO Web Search has roughly the same document count but only around 9,000 test queries with a single click-derived label each.

The queries come from real production traffic. Each search contains one primary query, which restates the user's information need, and zero or more support queries, which explore alternative phrasings, background information, or related entities. Each is scored independently. English makes up 65.8% of the set, followed by Spanish, Russian, German, French, and others, spanning domains like programming, law, health, finance, travel, and consumer goods.

The corpus is deliberately adversarial. Every document was considered a plausible result for at least one benchmark query, creating a dense set of difficult distractors: pages that match a query's topic or language but may miss a required date, entity, version, quantity, or aspect.

Three relevance sets instead of one ground truth

Rather than pick a single labeling source, Q2D-Web uses three complementary judgment sets to reduce false negatives and avoid biasing toward any one retrieval stack:

• Citation: documents an agent actually cited in a response. High precision, low recall.
• Web Ranking: up to 50 documents per query (43.1 on average), identified by an internal retrieval stack using BM25 and dense retrieval in the first stage, followed by cross-encoder reranking.
• Combined + LLM-Judged: union of the first two, plus LLM judgments over pooled results from BM25, ColBERTv2, and seven dense retrievers, adjudicated by a strict binary DeepSeek-V4-Flash judge.

The subsampling trick that makes it affordable

Running this benchmark end-to-end is brutal. A single evaluation on Q2D-Web with pplx-embed-v1-4b requires 4,608 H200 GPU-hours, and even a small model like EmbeddingGemma-300M requires nearly 200 H200 GPU-hours.

To make routine evaluation tractable, the team built a subsampled corpus using reciprocal rank fusion across construction retrievers. RRF-based sampling uses only 31.7% of the documents in the full corpus, preserves the full-corpus model ranking, and stays close to absolute scores, inflating mean Recall@1000 by only 4.5 points versus 11.1 points for random sampling at the same size. That cuts pplx-embed-v1-4b evaluation to about 1,500 GPU-hours.

Who wins, and where the ordering breaks

Thirteen retrievers were evaluated using Recall@1000 as the headline metric, on the theory that a first-stage retriever only needs to get relevant documents into the candidate pool before a reranker takes over. No single model sweeps the board. On Recall@1000, pplx-embed-v1-4b leads on Web Ranking (65.73) and Combined (69.11), while Nemotron-3-Embed-8B tops Citation (61.68). On Combined Recall@100 and nDCG@10 the order changes, with Nemotron-3-Embed-8B (30.03 and 47.44) above pplx-embed-v1-4b (29.82 and 45.84).

Scaling within families holds up cleanly: Qwen3-Embedding climbs from 57.89 at 0.6B to 64.53 at 8B on Combined Recall@1000, and Nemotron-3-Embed goes from 61.68 to 68.58 across the same jump. Perplexity flags an important caveat: their own models could benefit from an in-distribution advantage since the queries come from Perplexity traffic, even though the exact evaluation set was excluded from training.

How to get your model on the board

Submissions go through an evaluation request form pointing at a public Hugging Face repo. A few practical constraints worth knowing:

• Models are split into two brackets by parameter count: up to 1B and above 1B.
• Every submission runs on the subsampled corpus first. Only top-10 finishers in their bracket get the full-corpus evaluation.
• All queries and documents are truncated to 512 tokens with the model's own tokenizer, including instructions, prefixes, and special tokens, so a longer supported context confers no advantage.
• Models must load via standard transformers or sentence-transformers APIs. Anything needing trust_remote_code=True gets a slower manual review, and gated repos are ineligible.

For teams building agentic RAG pipelines, Q2D-Web is one of the few public references that tests embedding models under conditions that look like production: agent-rewritten queries, hard topical distractors, multilingual coverage, and label depth that does not silently punish good retrievers for surfacing correct-but-unjudged documents.