Microsoft's PrecisionCUA Teaches AI Agents to Fix Their Own Misclicks
Microsoft researchers show that letting vision-language models see their own mistakes as red crosses on screenshots more than doubles cursor-placement accuracy in code editors.
ProTakeaways
- Microsoft's PrecisionCUA shows red-cross visual feedback more than doubles Claude Opus 4.7's cursor accuracy from 22.8% to 48.1%.
- New 5,390-sample benchmark covers VS Code and Cursor, both themes, and character, word, and line targets.
- Refinement is capacity-gated: Qwen-3.5-9B stays near zero across all five turns regardless of prompting.
- Cross-editor gap exceeds 50 points: VS Code hits 90.2% while Cursor caps at 21.6% for the same model.
- Lightweight GTA1 finetune reaches 55.63% overall and beats GPT-5.4-Pro at every granularity.
- Specialized GUI models GUI-Actor and GUI-G2 collapse below 6% on sub-glyph cursor placement.
Computer use agents can click buttons and open menus reasonably well, but ask one to place the text cursor between the letters n and a inside a function name and it tends to miss by a few pixels, breaking the whole edit downstream. A new Microsoft technical report, PrecisionCUA (also circulated as See, Point, Refine), takes that failure mode seriously and asks a narrow but useful question: if you show a frontier vision-language model where its last click landed, can it correct itself?
The short answer is yes, provided the base model is already good enough. The team built a benchmark for sub-glyph cursor placement in VS Code and Cursor, then compared single-shot coordinate prediction against a closed loop where the model sees a red crosshair on its previous attempt and tries again. The gains are large for capable models and effectively zero for weaker ones.
Why clicking between two characters is hard
GUI grounding is the step where an agent turns an instruction into an (x, y) pixel on the screen. Small localization errors propagate into failed clicks, incorrect edits, and broken multi-step trajectories, making robust grounding the bridge between language understanding and executable interface control. Frontier models are increasingly reliable at coarse targets such as buttons, tabs, and icons, but remain substantially less reliable on editing-level actions that require fine-grained placement inside text-dense interfaces.
An IDE is a worst case for this. Line numbers, syntax-highlighted tokens, punctuation, and narrow caret boundaries create visually crowded targets with almost no tolerance for error. The authors frame the study around a concrete example: rename a function by placing the cursor between the n and the a inside function. Off by a few pixels and you edit the wrong token.
Point, look, point again
The method itself is deliberately simple, which is part of the point. The agent performs iterative grounding with one initial prediction followed by refinement turns. At turn one the model receives the instruction and the raw screenshot and predicts a coordinate. If it misses, the harness draws a red crosshair at that coordinate on a clean copy of the screenshot, appends the numerical value as Last attempt: [x, y], and asks the model to try again with a spatial-feedback template that tells it to reason about where the cross sits relative to the target.