Skip to main content
Back

OpenWebRL-4B Beats OpenAI's Web Agent Using Just 400 Training Examples

A multi-institutional research team releases OpenWebRL, an open framework that trains a 4B visual web agent to rival proprietary systems like OpenAI CUA using just 2.6K examples.

  • OpenWebRL is a new open framework for training visual web agents using online multi-turn RL directly on live websites, introduced in an arXiv paper by a multi-institutional team.
  • OpenWebRL-4B achieves 67.0% on Online-Mind2Web and 64.0% on DeepShop, rivaling proprietary systems like OpenAI CUA and Gemini CUA.
  • The model was trained with only 0.4K SFT trajectories and 2.2K RL tasks, dramatically reducing the data collection bottleneck of prior open agents.
  • The core algorithm is MM-GRPO, a multimodal multi-turn extension of GRPO that assigns rewards at the trajectory level rather than per action.
  • RL training measurably changes agent behavior: longer reasoning traces, blocker diagnosis, retry planning, and better hard-task generalization.
  • Training data, model weights, and code will be publicly released to support reproducible research.

Training a web agent that can actually navigate live websites has been one of the harder open problems in AI. The best-performing systems -- OpenAI's Computer Use Agent, Gemini CUA -- are proprietary black boxes. Open alternatives have historically leaned on massive collections of curated human demonstrations, which are expensive to gather and go stale fast. A new research paper from a multi-institutional team proposes a different path: skip the giant demonstration dataset, and instead let the agent learn by doing, directly on live websites, using reinforcement learning.

The paper introduces OpenWebRL, an end-to-end open framework for training visual web agents with online multi-turn RL. The result is OpenWebRL-4B, a 4-billion-parameter model that sets a new open-source state of the art on two of the hardest live-web benchmarks -- and does it with a remarkably small training footprint.

Why demonstrations alone hit a wall

Most open web agents today are trained through supervised fine-tuning (SFT) on recorded human trajectories: watch a person navigate a website, learn to imitate. The dominant approach relies on supervised post-training over large collections of curated web trajectories. The problem is structural: high-quality demonstrations are expensive to produce, and a static dataset can never cover the full diversity of the live web. Websites change layouts, add CAPTCHAs, and introduce new interaction patterns constantly.

Online RL sidesteps this by having the agent interact with real websites and learn from success or failure signals directly. Prior work like WebRL showed promise for text-based agents, but the visual dimension -- where the agent sees screenshots rather than raw HTML -- remained largely underexplored. OpenWebRL is the first framework to tackle the full visual pipeline end-to-end, on live websites, at scale.