Skip to main content
Back

Microsoft's ProRe Beats Standard AI Judges by 19.4% on GUI Tasks

A new research paper proposes replacing static trajectory judges with an active reasoner-actor system that probes the GUI to verify whether agent tasks actually succeeded.

  • ProRe is a proposed reward system for GUI agents that uses a reasoner-actor split instead of trajectory-only LLM-as-a-Judge.
  • A reasoner (Gemini-2.5-Pro) plans probing tasks; evaluator agents navigate the UI to collect additional state evidence.
  • Reports 93.7% average reward accuracy across AndroidWorld, AndroidLab, and MobileAgentBench, up 5.3 points over baselines.
  • Guiding test-time scaling with ProRe lifts M3A (GPT-4o) success rate by up to 22.4%.
  • Chain-of-claims reasoning tags policy-evaluator claim pairs as confirming, contradicting, complementary, or unrelated.
  • Code and prompts released at github.com/V-Droid-Agent/ProRe; per-task cost roughly $0.06.

Rewarding GUI agents is harder than it looks. When an agent taps through an Android app to book a flight or delete a file, how do you actually know it worked? A new arXiv paper from researchers at Microsoft Research, NTU, Tsinghua, and HKUST argues that the standard approach of pointing a large language model at a stack of screenshots and asking whether the task looks done misses too much of what actually happened on screen. Their proposed system, ProRe, replaces that passive review with something closer to an active audit.

Why LLM-as-a-Judge falls short on GUIs

Training and evaluating agents typically leans on one of two reward mechanisms. Rule-based verifiers use hand-written test scripts, which are accurate but hard to scale. AndroidWorld and WindowsAgentArena ship with more than 116 and 150 manually engineered unit tests to provide grounded completion signals, but authoring those scripts demands enough human effort to rule them out as a reward source for large-scale training. The alternative, LLM-as-a-Judge, prompts a model like GPT-4o with a trajectory of screenshots and asks whether the task succeeded.

The paper argues this fails for two reasons. GUI state is only partially visible in screenshots taken at fixed intervals, so critical evidence often never appears in the trajectory. General-purpose LLMs also lack the domain knowledge to interpret UI-specific behavior, such as recognizing that tapping