Skip to main content
Back

Microsoft's SentinelBench Forces AI Agents to Master the Art of Waiting

A new open-source benchmark tests whether AI agents can wait patiently for the right moment, rather than burning compute hammering refresh buttons

Paper, Full Access
  • Researchers from the University of Florida and Microsoft Research introduce SentinelBench, an open-source benchmark for long-running AI monitoring agents.
  • The benchmark contains 100 tasks across 10 synthetic web environments (email, calendar, finance, entertainment) with time-evolving state.
  • It measures three axes simultaneously: task completion, reaction time, and resource (token) use -- exposing the cost-vs-responsiveness tradeoff.
  • Results across 3 models and 2 browser-agent harnesses show agent design choices dramatically affect all three metrics.
  • Companion work (SentinelStep) showed smarter waiting strategies can lift 1-hour task success from 5.6% to 33.3%.
  • Full code, environments, and data pipelines are available at github.com/microsoft/sentinel_environments.

Most AI agent benchmarks reward speed and decisiveness: complete the task, call the tool, get the answer. But a growing class of real-world tasks demands something different -- the ability to wait. A research team from the University of Florida and Microsoft Research's AI Frontiers group has published a paper introducing SentinelBench, an open-source benchmark specifically designed to measure how well AI agents handle long-running monitoring tasks, where the right move is often to do nothing until the moment is right.

The problem with always-on agents

The default model of agent behavior is continuous action: issuing tool calls, refreshing pages, searching for alternatives, or otherwise trying to force progress. This is the wrong approach for many long-running tasks, which are better served by a strategy of sustained attention. Think of tasks like: "notify me when a flight price drops below $300," or "reply to that email once the meeting invite is accepted." An agent that constantly polls and re-checks burns through API tokens and compute budget, while an agent that sleeps too long misses the window to act.

Modern AI agents are increasingly capable of long and complex work. METR has tracked the 50% task-completion time horizon of AI models and found that it has doubled approximately every seven months, increasing from 4 seconds in 2019 to more than 16 hours in 2026. Yet almost no benchmarks exist to evaluate the specific skill of patient, event-driven monitoring -- the kind of behavior that underpins truly useful always-on assistants.

What SentinelBench actually tests

SentinelBench is a collection of 100 tasks divided across 10 synthetic web environments, including email, calendars, finance, professional networking, entertainment, and others. Each task scenario replays a sequence of events over time and asks the agent to complete work in this evolving context. The key design insight is that each environment is