Alibaba's OccuBench Tests AI Agents on Jobs No Benchmark Dared Touch
A new benchmark uses LLMs to simulate 65 professional domains, from ER triage to nuclear safety, and reveals every top model has blind spots.
PRO- OccuBench evaluates AI agents on 100 professional scenarios across 65 domains using LLMs as simulated environments.
- GPT-5.2 leads at 79.6%, but no model wins across all 10 industry categories tested.
- Implicit faults (truncated data, missing fields) cause bigger drops than explicit 500s or timeouts.
- GPT-5.2 improves 27.5 points from minimum to maximum reasoning effort on the benchmark.
- Strong agents are not strong simulators: GPT-5.2 makes the worst environment simulator despite topping agent rankings.
- Code, data, and paper released at GitHub and Hugging Face.
Most agent benchmarks live in a narrow slice of the world: browsers, terminals, GitHub repos, and a handful of retail APIs. A team from the Qwen group at Alibaba and CUHK argues this is exactly where AI agents matter least, and that the domains where they would be most valuable, from emergency triage to customs processing to nuclear safety monitoring, have no public environments to test against. Their answer is OccuBench, a benchmark that sidesteps the environment problem by letting an LLM play the role of the environment itself.
The untestable majority of professional work
Existing agent benchmarks are confined to domains with available environments: WebArena to web browsing, OSWorld to desktop operations, SWE-bench to code repositories, and TAU-bench to retail and airline APIs. The result is a severe evaluation blind spot covering most high-value professional work. The authors frame the gap bluntly: no benchmark can tell you whether an agent can triage a patient, process a customs declaration, or control greenhouse irrigation, because none of those environments exist in a public, testable form.
Two other issues compound the problem. Adding a new domain to something like WebArena means deploying real web apps; adding one to TAU-bench means integrating real APIs or writing simulators by hand. And no existing benchmark systematically tests what happens when the environment misbehaves, which is the default state of production systems.
Turning the LLM into the environment
If an LLM already knows how a hospital triage system or a network firewall works, it can generate the tool responses for that system on demand. Given a configuration containing a system prompt, tool schema, initial state, and state description, an LLM becomes a stateful, interactive environment simulator, called a Language Environment Simulator (LES). Environment construction becomes a configuration problem rather than an engineering one.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
