---
title: "DragOn Dataset Gives GUI Agents 3.5M Tasks to Finally Master Dragging"
subtitle: "DragOn fills a critical gap in GUI agent training data with 3.5M drag tasks spanning text highlighting, cell selection, resizing, and slider control"
url: "https://alphasignal.ai/news/dragon-dataset-gives-gui-agents-3-5m-tasks-to-finally-master-dragging"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-06-04T00:00:00.000Z"
updated: "2026-06-12T02:19:09.641Z"
type: "Paper"
topics:
  - "Agents"
  - "Benchmarks"
subtopics:
  - "Computer Use"
upvotes: 1162
source_url: "https://arxiv.org/abs/2606.06322"
comments: 0
truncated: true
---

# DragOn Dataset Gives GUI Agents 3.5M Tasks to Finally Master Dragging

> DragOn fills a critical gap in GUI agent training data with 3.5M drag tasks spanning text highlighting, cell selection, resizing, and slider control

GUI agents -- vision-language models that watch your screen and control your computer -- have made remarkable strides on one specific skill: clicking. But clicking is only half the story. A huge portion of real computer use involves dragging: selecting a block of text, resizing a spreadsheet column, scrubbing an audio slider, or drag-and-dropping a file. While million-scale datasets have enabled substantial progress on click-grounding, drag grounding data remains an order of magnitude smaller and current models fall short on complex drag-based interactions. A new paper introduces [DragOn](https://arxiv.org/abs/2606.06322), a benchmark and training dataset purpose-built to close that gap.

## The click-drag divide

To understand why this matters, you need to know what "grounding" means in this context. GUI grounding is the task of mapping a natural language instruction -- say, "highlight the second sentence" -- to precise pixel coordinates on screen. While existing grounding models achieve promising performance simulating the mouse click action on various click-based benchmarks, another essential mode of mouse interaction, namely dragging, remains largely underexplored.

The asymmetry is stark. The largest GUI visual grounding dataset to date covers 10M GUI elements over 1.3M GUI screenshots -- almost entirely click-focused. Prior drag-specific work like [GUI-Drag from OSU and Microsoft Research](https://arxiv.org/abs/2601.06031) introduced 161K text dragging examples and a benchmark with 5,333 examples, but that work was limited to text selection. Even Anthropic acknowledges that actions people perform effortlessly -- scrolling, dragging, zooming -- currently present challenges for Claude. The problem is real, widespread, and underserved by existing data.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [DragOn: A Benchmark and Dataset for Drag-Based GUI Interactions](https://arxiv.org/abs/2606.06322)
- [DragOn HTML Paper](https://arxiv.org/html/2606.06322v1)
- [Beyond Clicking: A Step Towards Generalist GUI Grounding via Text Dragging](https://arxiv.org/abs/2601.06031)
- [ShowUI-π: Flow-based Generative Models as GUI Dexterous Hands](https://arxiv.org/pdf/2512.24965)
- [Computer Use and GUI Agents in 2026: State of the Art](https://zylos.ai/research/2026-02-08-computer-use-gui-agents)
