Skip to main content
Back

Fujitsu and CMU's FieldWorkArena Exposes How AI Fails Real Factory Safety

Fujitsu and CMU release a benchmark of 886 tasks built from real factory, warehouse, and retail footage, exposing where today's multimodal agents break down.

  • Fujitsu and CMU released FieldWorkArena, a benchmark using real factory, warehouse, and retail footage.
  • 886 tasks span perception, decision-making, and multi-step combination workflows built from on-site interviews.
  • GPT-5.2 leads at 0.52 total accuracy versus a 0.74 human baseline on perception tasks.
  • New evaluator introduces Partially Correct scoring and a patched fuzzy_match for verbose agent outputs.
  • Spatial and long-video temporal reasoning remain major weaknesses across GPT and Gemini models.
  • Harness is Apache-2.0 on GitHub; dataset gated via Fujitsu portal request.

Most agent benchmarks live inside browsers, terminals, or synthetic worlds. A team from Fujitsu Research and Carnegie Mellon just released one that lives on the shop floor. FieldWorkArena is built entirely from footage recorded inside real factories, warehouses, convenience stores, and supermarkets, and it measures whether multimodal agents can actually spot PPE violations, gauge the distance between a forklift and a worker, or file an incident report.

Even the strongest current models are far from the human baseline. GPT-5.2 tops the leaderboard at 0.52 overall accuracy, while a human sample on perception tasks scored 0.74. Spatial reasoning from a single image and long-video temporal reasoning are where the wheels come off.

Why simulated benchmarks fell short

Agentic AI is being pitched hard for manufacturing, logistics, and retail, where labor shortages and safety monitoring are pressing problems. Yet there was no standard way to measure whether an agent could handle the messy input of a real workplace. Existing evaluations focus on simulated or digital environments, making it hard to verify robustness on actual field tasks, and few accept the mix of images, videos, and PDFs that a real site produces.

The Fujitsu and CMU team built the task set the hard way. Through interviews with site supervisors, consultants, and sales representatives, they identified safety and manufacturing near misses for factories and warehouses, plus employee compliance and customer behavior analysis for retail. Ground truth values for things like distance thresholds were pulled from on-site documentation rather than invented.

Inside the dataset

The corpus is small in file count but dense in tasks. Cameras were placed in a communications equipment factory, a warehouse handling electronic devices, and Japanese retail stores. Faces are blurred, and the retail scenes use actors performing realistic scenarios because customer consent could not be obtained. The benchmark ships 886 tasks across three categories: