Skip to main content
Back

MERRIN Benchmark Exposes AI Agents Scoring 40% Where Humans Hit 71%

A new UNC-led benchmark shows top search agents like Gemini 3.1 Pro top out at 40% accuracy, badly losing to humans on multimodal web queries.

  • MERRIN benchmarks search agents on 162 human-annotated multimodal web queries without modality cues (paper).
  • Best agent (Gemini 3.1 Pro, agentic search) hits 40.1%; average across all agents is just 22.3%.
  • Humans reach 71.4% with roughly 3x fewer searches and 20x higher URL precision.
  • Even with gold sources injected, accuracy only climbs to 47.7%, so reasoning is the main bottleneck.
  • Agents pull 87 to 96% of evidence from text, ignoring video and image sources they need.
  • Data and evaluation code available on GitHub, with encrypted answers to prevent contamination.

Search-augmented agents are getting a reality check. A new benchmark called MERRIN from researchers at UNC Chapel Hill, Virginia Tech, and UT Austin puts today's strongest agents through a gauntlet of realistic web queries that require pulling evidence from text, images, video, and audio and reasoning across them. Average accuracy across all agents is 22.3%, with the best performer reaching only 40.1%. Humans, meanwhile, hit 71.4% using far fewer searches.

Why existing benchmarks let agents off easy

Most multimodal search evaluations quietly hand the agent the answer to the hardest part of the problem: which modality to look at. Prior work often includes explicit modality cues, direct references to a specific source such as "In the following image...", limits the range of modalities to text and pictures while excluding video and audio, and leaves the noisy, conflicting nature of real-world web evidence underexplored in multimodal settings.

MERRIN throws all three crutches away. Questions are written in plain natural language, so the agent has to figure out on its own whether the answer lives in a chart, a lecture video, a podcast clip, or a Wikipedia paragraph. Evidence is drawn from the live, messy web rather than a curated corpus.

Diagram of MERRIN agent paths showing correct answers and three failure modes: reasoning error, modality error, and retrieval error

How the benchmark is built

MERRIN is small but hand-crafted. All 162 questions satisfy three criteria: natural text input without explicit references to specific modality sources, non-text modality required so correct answers strictly need visual, video, or audio evidence, and a single unambiguous short answer that enables automatic scoring. Questions were built and cross-reviewed by graduate and undergraduate annotators with NLP backgrounds, and roughly 40% of first-round candidates were rejected before revision.