Skip to main content
Back

Browser Use Shows Tool Design Beats Model Choice by 15 Points

Browser Use ran seven browser harnesses through 200 hard web tasks with one model, revealing a 15-point spread between the best and worst tool wrapping.

Repo, Full Access
  • Browser Use tested 7 browser harnesses on 200 hard tasks from BU Bench V2.1, same model and judge.
  • Six browser tools inside Codex scored 62.4 to 73.5; Browser Use 1.0 agent hit 77.5.
  • Tasks mostly drawn from Odysseys2, a CMU collab with 982 rubric checks across 100 long tasks.
  • Giving Browser Harness its own agent loop beat running it inside Codex by 4.6 points.
  • Benchmark is open MIT, runs on your own API keys via uv run python run_eval.py.
  • Harness choice now moves scores as much as model generation changes used to.

Browser Use compared seven browser harnesses across 200 long-horizon web tasks while holding the agent model, grading model, task set, and browser conditions constant. The surrounding agent software was the main variable. Scores ranged from 62.4 to 77.5, a 15.1-point spread that shows how strongly tool design and orchestration can affect browser-agent performance.

One model, a 15-point spread

The main comparison ran six browser tools inside Codex and compared them with Browser Use’s 1.0 agent. A separate paired test moved Browser Harness from Codex into its dedicated agent loop.

Results on BU Bench V2.1
Configuration Score Context
Six browser tools inside Codex 62.4 to 73.5 Main harness comparison
Browser Use 1.0 agent 77.5 Highest overall score
Browser Harness inside Codex 70.5 Paired agent-loop test
Browser Harness with its own loop 75.0 Paired agent-loop test

The comparison report estimates a 4.6-point lift for Browser Harness with its own loop, with a 95% confidence interval of 0.9 to 8.3 points. The displayed aggregate scores differ by 4.5 points because they are rounded. The confidence interval remains above zero, indicating a measurable advantage under this benchmark’s statistical analysis.

Two benchmarks, one demanding task mix

BU Bench V2.1 contains 200 tasks performed on live websites. Weighted rubrics award partial credit for completed requirements, allowing the benchmark to distinguish between partial and complete results. The grading model, called the judge, evaluates each agent’s output against those rubrics.

The default configuration runs all 200 tasks with OpenAI GPT-6 Luna xhigh as the agent model and GPT-5.6 Luna xhigh as the judge. It uses Browser Use Cloud, allows 3,600 seconds per task, and supports up to 100 concurrent tasks. The harness comparison used GPT-6.1 Terra medium and kept the remaining evaluation conditions fixed.

Most tasks come from Odysseys2, a Browser Use and Carnegie Mellon benchmark with 100 live-web tasks and 982 pass/fail checks. Each task contains 8 to 12 checks, and full completion requires satisfying every check. Instructions have a median length of 271 words and commonly request several related deliverables across multiple pages.

Odysseys2’s strongest reported agent, GPT-6 Astra at default effort, achieved a mean rubric score of 84.6 while failing to complete every check on 53 of the 100 tasks. That gap between partial-credit scores and strict completion rates explains why the benchmark reports both detailed rubric performance and task-level success.

The outer loop earns 4.6 points

A browser harness determines what the model sees, which browser operations it can call, how state persists between steps, and how the agent recovers from failed actions. The evaluated approaches range from Chrome DevTools MCP, which exposes 32 operations through the Model Context Protocol, to Browser Use’s smaller tool surface.

Inside Codex’s general-purpose agent loop, Browser Harness showed no statistically significant score difference from Playwright MCP. Their all-in costs were $0.73 and $0.79 per task, respectively.

Moving Browser Harness into its dedicated loop raised its score from 70.5 to 75.0. The result isolates the effect of orchestration around the browser tools: planning, state management, retry behavior, and decisions about which action to take next. Developers evaluating browser stacks therefore need to test the tool API and the agent loop as a combined system.

Reproduce the evaluation

Developers need Git, uv, an OpenAI API key, and a Browser Use Cloud key. The following commands pin the repository to V2.1.1 and start the default evaluation:

git clone --branch v2.1.1 --single-branch https://github.com/browser-use/benchmark.git
cd benchmark

export OPENAI_API_KEY="..."
export BROWSER_USE_API_KEY="..."

uv run python run_eval.py

The runner loads the full V2.1 task set, applies weighted rubrics, and saves traces and screenshots for failure analysis. It supports cloud and local browsers and can run selected task IDs for faster debugging. Full evaluations require paid model and browser usage, and 100 concurrent long-running tasks can consume credits quickly.

Keep every comparison controlled

Reliable harness comparisons require the same model, effort setting, judge, task revision, browser backend, network environment, timeout, and concurrency policy. Browser connections should also use the same Chrome DevTools Protocol, or CDP, configuration. Local Chrome and cloud stealth-browser results should be reported as separate experimental conditions.

  1. Pin the benchmark release and record every task ID.
  2. Fix the agent model, judge, effort level, and prompts.
  3. Use one browser backend and CDP configuration.
  4. Vary only the harness or agent loop under evaluation.
  5. Report aggregate scores, confidence intervals, cost per task, and failures.
  6. Inspect saved traces before attributing failures to the model.

The V2.1.1 release includes CAPTCHA and source-verification corrections, along with a redesigned playback for Reuters task 185. All 200 task IDs remain stable. Pinning the revision matters because changes to live websites, grading logic, or task playback can shift scores independently of agent quality.

Encrypted task data reduces straightforward exposure to model providers, although it cannot establish that every underlying web page is absent from training data. The benchmark is best suited to controlled comparisons within one pinned environment.

On BU Bench V2.1, the surrounding browser-agent software accounted for a 15.1-point range with the model fixed, while a dedicated loop added an estimated 4.6 points for one harness. Generalizing those results to other models, websites, and browser backends requires separate runs under equally controlled conditions.