TurtleAI Exposes How Top AI Models Fail Basic Geometry Drawing Tasks
TurtleAI exposes a critical blind spot in today's best vision-language models: even GPT-5 and o3 fail to translate geometric images into working Python code more than 60% of the time.
- TurtleAI is a new benchmark of 823 tasks testing whether VLMs can write Python Turtle Graphics code to reproduce geometric images.
- All 20+ models tested fail badly: o3 leads at 40.2%, GPT-4o hits 26.5%, and most open-source models score near zero.
- Failure analysis shows spatial reasoning is the #1 bottleneck for GPT-4o (26.5% of failures), while Qwen2-VL-72B mainly fails to translate correct visual reasoning into working code (30.4%).
- TurtleAI-Datagen generates 738k training pairs from just 10 seed examples using reference-guided code mutation, elite selection, and CoT labeling.
- Fine-tuning Qwen2-VL-72B on this data raises its success rate from 11.8% to 35.3%, but spatial reasoning errors remain unchanged and general coding ability degrades.
- The benchmark, datasets, and generation scripts are being released publicly to support future research in visual programming and spatial reasoning.
Vision-language models can caption photos, answer questions about charts, and write code from natural language. But hand them a picture of a geometric pattern and ask them to write the Python code that draws it, and they fall apart. That is the uncomfortable finding at the center of TurtleAI, a new benchmark that stress-tests multimodal models on education-oriented visual programming tasks using Python's Turtle Graphics library.
Turtle Graphics is the classic K-12 programming environment where a virtual "turtle" moves around a canvas drawing lines and shapes in response to commands like forward(), left(), and begin_fill(). It is deliberately simple, which makes it a clean testbed: if a model cannot write correct turtle code from a picture of a hexagon, that is a meaningful signal about its spatial reasoning and code-synthesis abilities.
A benchmark built from real classrooms
The benchmark contains 823 tasks drawn from three sources. TurtleAI-DSBasic (102 tasks) comes directly from XLogoOnline, a visual programming platform used by tens of thousands of students. TurtleAI-DSCraft (102 tasks) uses the same patterns but replaces the clean digital images with hand-drawn versions, testing robustness to real-world input variation. TurtleAI-DSSyn (619 tasks) is a synthetically generated set of harder problems. Each task is a pair: a target image and an instruction asking the model to produce a draw(t) Python function that faithfully reproduces it.
Tasks are organized into six geometric categories and three difficulty levels. The categories span basic shapes all the way up to composite patterns that combine rotation, scaling, and translation simultaneously. Difficulty is rated Easy, Medium, or Hard based on how many transformations are involved and how precisely they must be reproduced.
Every model struggles, including the frontier ones
Evaluation of 20 VLMs reveals that state-of-the-art models like GPT-4o and Qwen2-VL-72B struggle with these tasks, achieving success rates of only 26.5% and 11.8% respectively. The full leaderboard tells an even starker story:
- o3 leads all base models at 40.2% on the basic dataset, dropping to 13.7% on the full benchmark.
- o4-mini reaches 36.3% on basic tasks and 15.9% on the full set.
- GPT-5 (medium reasoning effort)
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
