Tongji University's PlanBench-V Exposes Where Every AI Still Fails Urban Planning
PlanBench-V stress-tests 17 VLMs on urban planning maps, revealing a 27% generational leap but a hard ceiling on policy-aware decision-making
Takeaways
- First professional benchmark: PlanBench-V introduces 223 expert-annotated planning maps and 1,629 QA pairs for evaluating VLMs on urban planning tasks.
- Four-level PRAI framework: Evaluates models on Perception, Reasoning, Association, and Implementation -- mirroring the cognitive pipeline of professional planners.
- 27% generational leap: Qwen3.6-Plus (2026) scores 1.701 vs GPT-4o's (2025) 1.342 overall, with every 2026 model beating the best 2025 score.
- Hard ceiling on policy judgment: All 17 tested models fail at Implementation tasks requiring evaluative judgment, policy sensitivity, and constraint-aware decisions.
- Domain fine-tuning works: The companion PlanGPT-VL 7B model matches 72B general-purpose models on planning tasks after domain-specific training.
- Open dataset and code: Full benchmark, SPMD dataset, and evaluation scripts available at plangpt.github.io.
Urban planning maps are not like ordinary images. They encode land-use regulations, zoning boundaries, infrastructure layouts, and policy constraints into dense, symbol-laden visuals that take trained professionals years to read fluently. Now, a team from Tongji University and Peking University has built the first rigorous benchmark to find out just how well today's Vision-Language Models (VLMs) can handle them -- and the results are a mixed bag of genuine progress and stubborn failure.
PlanBench-V is a domain-specific evaluation suite that goes far beyond asking a model to name what it sees on a map. It probes a full four-stage cognitive pipeline modeled on how professional planners actually think, and it tests 17 models across two generations of AI. The headline finding: the best 2026 model beats the best 2025 model by 27%, but every single model still falls apart when asked to make real planning decisions.
Why general benchmarks miss the point
Spatial planning maps are central to territorial governance, translating planning objectives, regulations, and spatial strategies into visual forms for decision-making and institutional coordination -- but interpreting them requires fine-grained visual perception, spatial reasoning, and policy-informed professional judgment, creating major challenges for both human learners and AI systems. That combination is what makes them uniquely hard.
With the rapid progress of VLMs, their use in urban planning analysis is gaining attention, yet existing multimodal benchmarks mainly target general visual understanding and overlook the domain-specific cognitive processes of planning practice. Benchmarks like MMMU or general VQA suites test whether a model can describe a chart or identify an object. They do not test whether a model can read a zoning map, infer what regulations apply, and then flag a proposed development that violates setback rules.
This gap matters because the urban planning sector is actively exploring AI-assisted workflows. PlanGPT-VL, the first domain-specific VLM designed for urban planning map interpretation, was built precisely to bridge the gap between general multimodal AI and professional spatial analysis, enabling planners, policymakers, and educators to understand, evaluate, and reason over urban planning maps with expert-level precision. But without a rigorous benchmark, there was no principled way to measure whether any model was actually up to the task.