Lenovo's AdaZoom-GUI Beats Larger Models at Clicking Tiny 4K Screen Icons
A Lenovo and Tsinghua team built a GUI grounding pipeline that rewrites ambiguous instructions and zooms in only when the target is small.
ProTakeaways
- AdaZoom-GUI splits GUI grounding into instruction refinement plus a small specialized grounding model.
- Conditional zoom-in triggers a second inference pass only when the predicted bounding box is small.
- The 4B model is trained with GRPO using point-in-box and IoU rewards to predict click plus box.
- Reaches 76.8 on ScreenSpot-Pro, beating GUI specialists with larger parameter counts.
- Unconditional zoom hurts on low-res ScreenSpot-v2 (0.916); conditional zoom improves it (0.945).
- Built on Qwen3-VL-4B-Instruct with a Qwen3.5-397B refinement model; no code released yet.
Getting a vision-language model to reliably click the right button on a 4K screenshot is harder than it sounds. Icons are tiny, resolutions are enormous, and user instructions like launch a new file explorer rarely match anything literally written on screen. A team from Lenovo Research and Tsinghua University is proposing a fix called AdaZoom-GUI, a framework that splits the problem into two smaller ones and only pays the cost of a zoom-in pass when it is actually needed.
On the ScreenSpot-Pro benchmark of high-resolution professional GUIs, the 4B-parameter model reaches an average score of 76.8, surpassing GUI-specialist models with comparable or larger parameter counts.
Why GUI grounding keeps breaking
GUI grounding asks a vision-language model to locate the target element on a screenshot given a natural language instruction, and it stays hard because screenshots are high resolution, UI elements are small, and user instructions are ambiguous. Two failure modes dominate current systems:
- Downsampling a 4K screenshot to fit a VLM's input resolution shrinks a 20-pixel icon into near-nothing.
- Fixed two-stage pipelines that always crop and re-run inference waste compute on easy screens and can hurt accuracy by throwing away surrounding context.
The authors argue there is also a subtler problem. Instruction comprehension and pixel localization are different skills, and training one model to do both on datasets full of simple commands teaches neither well.