AA-3D ranks AI models by the 3D scenes they build

Most AI benchmarks score answers to math problems or completed coding tasks. Artificial Analysis has introduced AA-3D, which tests whether 21 leading models can turn written briefs into interactive three.js web pages. Blind human comparisons determine the benchmark’s Elo rankings.

Agents build, inspect, and revise

AA-3D v1.0 contains 55 tasks across seven domains, including landmarks, machines, living bodies, games, and space. Each brief specifies required visual elements and interactions. The output is an index.html page built with three.js, a JavaScript library for rendering 3D graphics in a browser.

Each model operates as an agent inside an offline sandbox, where it can write code, render the scene, inspect screenshots, and revise its work. The environment includes Blender, Rapier physics, Python, a headless browser, and a library of CC0 assets. Runs end after three hours or 250 agent turns.

Evaluators see completed scenes in pairs and spend at least 30 seconds exploring them before voting. One comparison in four checks voter attention. More than 10,000 accepted votes from over 1,000 panelists feed a Bradley-Terry model, which estimates relative win rates and converts them into Elo scores with 95% confidence intervals.

Four models share the top tier

Claude Sonnet 5.5 holds the highest nominal score at 1194 Elo. The report treats the first four models as statistically tied because their 95% confidence intervals overlap with Claude Sonnet 5.5’s interval. Anthropic and OpenAI supply all four.

The table shows selected entries, so ranks 6, 7, 9, and 10 are omitted. Among models with openly available weights, Kimi K3 ranks 11th at 982 Elo and MiMo-V2.6-Pro ranks 12th at 977, leaving both more than 200 points behind the nominal leader. Thinking Machines’ Inkling and NVIDIA’s Nemotron 3 Ultra score below 580 Elo.

Early visual feedback tracks higher scores

The strongest relationship for agent developers concerns when models inspect their own output. The top five models first examine a rendered image after a median of 10 to 13 turns, and they do so by turn 20 in at least 89% of their scenes. Models ranked 14th through 20th wait a median of 15 to 40 turns. Among the 19 image-capable models, earlier inspection correlates strongly with Elo, with a Spearman coefficient of 0.82.

MiniMax-M3 illustrates why timing and frequency capture different behavior. It inspects its scenes 54 times per run, the highest rate in the benchmark, yet reaches its first inspection by turn 20 in only 7% of scenes and ranks 19th. Image access also appears relevant: text-only GLM-5.3 scores 197 Elo below the trend line, while image-capable GLM-5.3-Flash scores 149 points higher despite a lower Artificial Analysis Intelligence Index.

Across all 21 models, file reading, token output, turn count, and interface controls show different relationships with Elo:

• File reading: The top five read files and documentation 9.4 times per scene, compared with 18.7 for the remaining models. More reading correlates with lower Elo across the full set, with a Pearson coefficient of -0.74.
• Output volume: Output tokens have a weak, statistically insignificant relationship with Elo, with a Spearman coefficient of 0.33.
• Run length: Turn count also has a weak, statistically insignificant relationship with Elo, with a Spearman coefficient of -0.27.
• Interactive controls: The top five produce a median of 15 to 20 buttons and other controls, compared with 6 to 8 for the bottom three. Control count correlates strongly with Elo, with a Pearson coefficient of 0.90.

A 160-fold cost range

Reported model cost ranges from $0.20 to $32.08 per scene. Input accounts for 69% of spending because agents repeatedly send their accumulated context during long runs. Prompt caching can therefore reduce costs substantially when a provider supports it.

Price and benchmark score diverge sharply in several cases. GPT-6 Luna costs $0.20 per scene and ranks eighth, while Kimi K3 spends 68 times as much for a similar Elo. GPT-6 Sol occupies a strong point on the benchmark’s cost-quality frontier, scoring 1123 Elo at $3.46 per scene. Claude Sonnet 5.5 posts the highest nominal score at about $15 per run.

General reasoning explains much of the ranking

Scores on the Artificial Analysis Intelligence Index correlate closely with AA-3D performance. Each additional index point corresponds to about 18 Elo, with a Pearson coefficient of 0.92. That relationship is stronger than AA-3D’s correlation with Terminal-Bench 4.0 at 0.81 or SciCode at 0.76.

Several OpenAI models outperform the index-based prediction. GPT-6 Luna finishes 141 Elo above its predicted score, GPT-6 Sol finishes 93 points above, and GPT-6 Astra finishes 46 points above. Those residuals indicate relative strength on this combination of spatial reasoning, coding, and visual iteration, while the benchmark provides no evidence about which training choices produced it.

Visible detail reflects model strength

The launch report uses screenshot file size and empty image area as proxies for visible detail. Within each task, the most detailed fifth of scenes has screenshots averaging 168 KB, compared with 84 KB for the least detailed fifth, and 22% empty area, compared with 60%. The detailed group wins 63% of its votes, while the sparse group wins 31%.

Most of that difference appears between models. Within a single model’s outputs, a scene one standard deviation above that model’s average detail wins about 1.5 percentage points more often. Visual density therefore identifies stronger outputs across the leaderboard more reliably than it offers a direct recipe for improving one model’s scenes.

Practical lessons for agent builders

AA-3D gives developers several concrete hypotheses to test in coding agents, generative interfaces, three.js tools, and other systems that create visual artifacts through multiple turns:

• Give the agent access to rendered output and schedule its first inspection early.
• Let the agent revise against screenshots instead of relying only on source code and logs.
• Cache repeated prompt context where model APIs and workload design allow it.
• Measure useful interaction controls alongside visual quality.
• Evaluate added turns and tokens empirically because longer runs showed no reliable score advantage.

What the scores leave open

• Human preference has a defined scope: The votes measure perceived visual quality and interaction during brief exploration. Correctness, accessibility, security, maintainability, and runtime performance require separate evaluation.
• Elo is relative: Scores depend on the included models, tasks, voting panel, and comparison set. They are useful within AA-3D v1.0 rather than as absolute quality grades.
• Correlations are observational: Early image inspection, control count, and other behaviors may accompany stronger models without independently causing higher scores.
• The environment shapes the result: Three.js, the offline sandbox, available CC0 assets, and the three-hour or 250-turn limit may favor different capabilities than a production workflow.

For teams selecting a model under this benchmark’s conditions, Claude Sonnet 5.5 and Claude Opus 5.5 sit in the statistically tied top group. GPT-6 Sol offers a lower-cost option close to the leaders, while GPT-6 Luna provides the cheapest result among the selected high-ranking entries.

Artificial Analysis publishes the generated scenes and turn-by-turn agent runs for inspection. Visitors can also submit blind comparisons through the voting page. The underlying agent harness, Stirrup, is open source.