Zhejiang University's UI-Zoomer Boosts AI Click Accuracy by 13.4% Without Training
A training-free framework that only zooms into UI screenshots when the model is uncertain, boosting grounding accuracy by up to 13.4 percent.
Takeaways
- UI-Zoomer is a training-free zoom-in framework for GUI grounding that only triggers when the model is uncertain
- Uses spatial agreement across N=8 stochastic samples plus token confidence as a reliability gate
- Crop size is derived from variance decomposition into inter-sample spread and intra-sample box extent
- Gains of up to +13.4% on ScreenSpot-Pro, +10.3% on UI-Vision, +4.2% on ScreenSpot-v2
- Works across Qwen2.5-VL-7B, GUI-G2-7B, UI-Venus-7B and UI-Venus-72B with no retraining
- Code available at github.com/ZJU-REAL/UI-Zoomer
Click-a-button agents keep tripping over the same problem: they can describe what they see, but they cannot reliably tell you which pixel to click when the target is a 12x12 icon buried in a 4K Photoshop panel. A new paper out of Zhejiang University and Ant Group proposes a fix that requires no training, no new model weights, and no extra data. It is called UI-Zoomer, and it treats the decision of whether to zoom into a screenshot, and by how much, as an uncertainty estimation problem.
On the hardest professional-desktop benchmark, adding UI-Zoomer on top of existing GUI models pushes accuracy up by as much as 13.4 percentage points without changing a single model parameter.
Why current GUI models keep missing tiny targets
GUI grounding means turning an instruction like "click the eyedropper tool" into pixel coordinates on a screenshot. Modern vision-language models handle desktop and mobile interfaces reasonably well, but they collapse on professional software. On ScreenSpot-Pro, a benchmark of 1,581 expert-annotated instructions sourced from 23 applications across five industries and three operating systems, existing models achieve only around 18.9% accuracy. The reason is brutal: target UI elements often constitute only 0.07% of the total image area, compared to 2.01% in mainstream benchmarks.
The standard workaround is test-time zoom-in: crop the screenshot to a smaller region, feed it back into the model at higher effective resolution, and re-predict. It works, but existing crop-based methods share two problems. They crop indiscriminately, running the expensive second pass on every single instance, and they use a fixed crop ratio that ignores whether the model is scattered across the whole screen or already narrowed in on the right area. The authors show empirically that unconditional zoom-in on ScreenSpot-v2 actually degrades accuracy below the direct-prediction baseline while blowing up latency from about 36 minutes to nearly seven hours.