Skip to main content
Back

Shanghai AI Lab's APB Exposes GPT-5's Hidden Planning Gaps in Agents

APB stress-tests 12 frontier models on planning-only tasks, exposing critical weaknesses in long-horizon reasoning, tool noise, and knowing when to refuse

Paper, Full Access
  • New benchmark: APB introduces 4,209 multimodal planning tasks across 22 domains to diagnose LLM agent planning failures separately from execution failures.
  • Five stress tests: Holistic planning, step-wise planning, extraneous tools, broken tools, and unsolvable tasks each expose distinct planning weaknesses.
  • Frontier model results: GPT-5 leads holistic planning at 74.5% correctness; best open-source model (Qwen3-VL-32B) scores only 22.5%.
  • Tool noise is critical: Adding irrelevant distractor tools drops GPT-5 holistic accuracy by 22% and Claude Sonnet 4.5 by 12.3%; step-wise planning is far more resilient.
  • Refinement works selectively: Critic-guided refinement with error taxonomy boosts holistic planning dramatically (22% to 60% on InternVL3.5-241B) but hurts step-wise planning.
  • Execution validated: APB-guided plan refinement improves downstream task reward on ToolSandbox and τ²-bench by up to 7 points across three tested models.

When an LLM agent fails a task, was it a bad plan or a bad execution? Today's benchmarks can't tell you. They measure whether the agent crossed the finish line, not whether it mapped the right route. A new paper from researchers at Shanghai AI Laboratory, Tongji University, and collaborating institutions introduces the Agent Planning Benchmark (APB), a diagnostic framework designed to isolate planning quality from execution noise, and the results reveal uncomfortable gaps in even the most capable frontier models.

The blind spot in agent evaluation

Planning is central to LLM agents: before acting, an agent must decompose goals, select tools, reason over constraints, and decide when a task is infeasible. Yet existing agent evaluations often report only end-to-end success, making it difficult to determine whether failures stem from planning or execution. This is a real problem in production: a model can make every individual tool call look reasonable while still assembling a globally incoherent plan, or it can produce a plausible-looking plan that quietly violates a constraint buried in the system prompt.

The distinction matters because the fix is different. Bad planning calls for better reasoning at the strategy layer. Bad execution calls for better tool integration or environment handling. Without separating the two, you're debugging in the dark.

What APB actually tests

APB is a planning-specific diagnostic benchmark with 4,209 multimodal cases across 22 domains and five settings, covering holistic planning, feedback-conditioned step-wise planning, and robustness under extraneous tools, broken tools, and unsolvable tasks. The five task types are designed to stress different planning sub-skills:

  • Holistic Planning (1,109 instances): Given a query and a tool set, produce a complete plan and ordered tool chain in a single pass , no feedback, no retries. This tests whether a model can maintain global consistency across a long trajectory before seeing any execution results.
  • Step-wise Planning (900 instances): Given a partially executed trajectory and prior tool outputs, predict the next 1, 2, or 3 actions. This tests feedback-conditioned local reasoning.
  • Tool-Extraneous Planning (1,500 instances): The tool set is polluted with semantically similar but functionally useless distractor tools (2 to 10 extras). Can the model ignore them?
  • Tool-Broken Planning (300 instances): A critical tool returns an error. A replacement tool with a different name is available. Does the model switch, retry, or give up?
  • Unsolvable Planning (400 instances): Tasks are made logically impossible via contradictory constraints, missing information, removed tools, or absent visual context. The correct answer is to refuse.