Microsoft's ThinkingBox Catches AI Agents Lying About Database Changes

Anyone who has shipped an LLM agent into production knows the sinking feeling of reading a transcript that says request completed successfully and then discovering the underlying database never changed. Microsoft's new open-source project ThinkingBox is built to catch that class of silent failure, grading agents on the side effects they leave behind rather than the words they say.

The release comes in two parts. microsoft/thinkingbox is the runtime and CLI, while microsoft/thinkingbox-data holds the benchmark tasks, synthetic records, and MCP servers. Both are MIT-licensed. The framework runs on Linux (including WSL), targets Python 3.12, and installs with uv or pip. Everything runs locally; your only cost is the LLM API calls the harness makes to whichever model you're evaluating.

Grade the database, not the transcript

The motivating example in the writeup is a travel agent asked to add a quiet-room preference to a hotel booking. The agent confirms the request was added, the transcript looks clean, but the booking's special_requests field is still empty. A judge grading only the final response would pass that run.

ThinkingBox builds each task around four connected pieces:

• A known initial state and expected outcome, so the evaluator can diff actual records against a golden database hash.
• A controlled tool surface exposed as MCP servers, so the agent can only mutate state through instrumented calls.
• A simulated user (a separate LLM) that only reveals details like booking references when explicitly asked.
• A clean environment per attempt, so one failed run cannot poison the next.

Tests assert on the records left behind rather than the sequence of tool calls, so any trajectory that produces the required outcome passes. For fuzzy requirements like "the response should mention the preference is subject to availability," ThinkingBox falls back to a narrow LLM judge question.

How the runtime is wired

The MCP Session Proxy is a long-running HTTP server that fronts a fleet of MCP tool processes, owning the lifecycle of each run and invoking three reserved tools for initialization, effect collection, and teardown that the agent never sees. When you call tb infer, the CLI creates an isolated session with a fresh set of MCP processes, runs the agent-user conversation, collects the effects, executes the Python assertions, and tears everything down.

Runs can be batched concurrently, with every attempt getting its own session ID and freshly initialized MCP processes even when several run in parallel. The agent and simulated user use separately configurable LLM endpoints, so you can swap in a new model under test without touching task definitions.

507 tasks across five domains

ThinkingBox-Bench ships 507 realistic multi-turn tasks across five business domains, built in partnership with Toloka. The distribution:

• Retail and e-commerce: 98 tasks (returns, refunds, warranties, promotions)
• Travel and hospitality: 104 tasks (bookings, cancellations, invoices)
• Auto insurance: 100 tasks (billing, claims, driver changes)
• Neobank internal IT: 104 tasks (identity, access, devices)
• Consulting IT and HR: 101 tasks (onboarding, provisioning, assets)

Among peers, tau-bench and tau2-bench are the closest comparisons, since they also cover stateful environments, controlled tool surfaces, interactive users, and isolated repeatable runs. ThinkingBox's differentiator is the MCP-native lifecycle plus the domain breadth.

Reliability is not capability

Microsoft ran 12 proprietary and open-weight models with 20 trials per task. Frontier models dominate pass@1, but the more interesting numbers are pass@20 (did any of 20 attempts succeed) versus pass^20 (did all 20 succeed).

The top model reached 91.12% pass@20 but only 25.25% pass^20, meaning it found a working trajectory on most tasks yet could rarely repeat it consistently across all 20 attempts. Two mid-tier models with nearly identical pass@1 scores looked completely different under this lens: Claude Opus 4.6 reached 70.02% pass@20 and 13.81% pass^20, while Kimi-K2.6 hit 84.22% pass@20 but only 3.16% pass^20. One finds solutions on more tasks; the other repeats them more reliably. For anyone deciding where to add retries, guardrails, or human review, that distinction matters more than a single leaderboard number.

Where agents actually fail

The failure taxonomy from analyzing traces is worth internalizing:

Failure category	Share of failed traces
Tool usage errors and failed recoveries	77.5%
Wrong state change (mutation succeeds but with wrong values)	12.1%
Response quality (incomplete or premature)	7.9%
Missing state change (lookups only, no mutation)	2.5%

Tool errors and unsuccessful lookups account for 77.5% of failed traces, meaning most agents get far enough to attempt the workflow and then fail during execution. The implication for anyone building agent loops: interpret every tool result, replan after failures, and verify state before reporting success.

Who should care

If you're building tool-using agents for customer support, IT ops, or any workflow where the truth lives in a database rather than a chat window, ThinkingBox gives you a reusable harness for the evaluation work that trajectory-based benchmarks skip. The framework is also explicitly designed for generating offline training data and for use inside reinforcement learning loops, which fits Microsoft's stated origin as an internal tool for the Copilot Studio RL team. Start with the tutorial if you want to add your own MCP server and scenario before touching the 507-task benchmark.