Skip to main content
Back

DragOn Dataset Gives GUI Agents 3.5M Tasks to Finally Master Dragging

DragOn fills a critical gap in GUI agent training data with 3.5M drag tasks spanning text highlighting, cell selection, resizing, and slider control

Paper, Full Access
  • The gap: Drag grounding data is an order of magnitude smaller than click data, and all major models (GPT, Claude, Qwen) perform poorly on drag tasks.
  • DragOn dataset: 286K training screenshots and 3.5M tasks across text highlighting, cell selection, element resizing, and slider manipulation.
  • Evaluation suite: A held-out 2,000-example benchmark tests generalization across all four drag domains.
  • Model results: A Qwen VLM fine-tuned on DragOn shows improved performance on downstream computer-use tasks vs. the base model.
  • Context: Prior drag datasets like GUI-Drag topped out at 161K examples and covered only text selection; DragOn is 20x larger and covers four distinct interaction types.
  • Limitations: Synthetic data pipeline may not cover all real-world drag diversity; no trajectory modeling between start and end points.

GUI agents -- vision-language models that watch your screen and control your computer -- have made remarkable strides on one specific skill: clicking. But clicking is only half the story. A huge portion of real computer use involves dragging: selecting a block of text, resizing a spreadsheet column, scrubbing an audio slider, or drag-and-dropping a file. While million-scale datasets have enabled substantial progress on click-grounding, drag grounding data remains an order of magnitude smaller and current models fall short on complex drag-based interactions. A new paper introduces DragOn, a benchmark and training dataset purpose-built to close that gap.

The click-drag divide

To understand why this matters, you need to know what "grounding" means in this context. GUI grounding is the task of mapping a natural language instruction -- say, "highlight the second sentence" -- to precise pixel coordinates on screen. While existing grounding models achieve promising performance simulating the mouse click action on various click-based benchmarks, another essential mode of mouse interaction, namely dragging, remains largely underexplored.

The asymmetry is stark. The largest GUI visual grounding dataset to date covers 10M GUI elements over 1.3M GUI screenshots -- almost entirely click-focused. Prior drag-specific work like GUI-Drag from OSU and Microsoft Research introduced 161K text dragging examples and a benchmark with 5,333 examples, but that work was limited to text selection. Even Anthropic acknowledges that actions people perform effortlessly -- scrolling, dragging, zooming -- currently present challenges for Claude. The problem is real, widespread, and underserved by existing data.