Zhejiang University's MAS-Bench Shows Hybrid Phone Agents Beat Tap-Only AI by 39%
A new mobile agent benchmark shows mixing GUI taps with API calls and deep-links lifts task success from 35% to 68%.
Takeaways
- MAS-Bench introduces the first benchmark for hybrid GUI plus shortcut mobile agents, accepted to ACL 2026. Paper
- Includes 139 tasks across 11 apps, 88 predefined shortcuts (APIs, deep-links, RPA scripts), and 9 evaluation metrics.
- MAS-GLM-4.5V reaches 68.3% success; MAS-MobileAgent jumps from 35.2% to 63.3%, an 80% relative gain.
- Hybrid agents cut successful-task step ratio by 38.9%, executing tasks in fewer steps than human experts.
- Stress tests expose shortcut misuse: weak models drop 66.7% success under irrelevant shortcut distractors.
- Agent-generated shortcuts still lag human-designed ones; coordinate-free dynamic shortcuts show most promise. Project page
Mobile agents that drive your phone by tapping and swiping through the interface are flexible, but painfully slow. A new benchmark called MAS-Bench, out of Zhejiang University and vivo AI Lab, argues for hybrid agents: ones that call APIs, deep-links, and pre-built automation scripts when they can, and fall back to GUI actions only when they must. The team reports hybrid agents reaching 68.3% success and running 39% more efficiently than GUI-only counterparts.
Why tap-and-swipe agents keep hitting a wall
Most mobile agent research so far has focused on making models act like humans, tapping through screens one action at a time. That yields generality at a steep cost. Step-by-step execution of routine tasks is slow and expensive, and in complex operations, model hallucinations or misinterpretations compound into cascading errors that drag down success rates.
Phones already expose faster paths to the same functionality. A hybrid agent can skip multi-step navigation by firing a single shortcut, collapsing operational complexity and time. What was missing is a rigorous way to measure whether agents can pick the right shortcut at the right moment, or generate new ones from experience.
What MAS-Bench actually contains
MAS-Bench, accepted to ACL 2026, runs on an Android emulator with snapshot-based resets and dedicated test accounts, so evaluations are reproducible without leaking into real user data. It ships with: