Artificial Analysis' AA-Robotics Puts GPT-6 Astra in Control of a Real Robot Arm
Artificial Analysis launched AA-Robotics v0.1, a research preview that puts frontier LLMs in charge of a 6-axis robot arm across eight tabletop tasks.
Takeaways
- Artificial Analysis launched AA-Robotics v0.1, a benchmark giving LLMs control of a 6-axis robot arm.
- GPT-6 Astra tops the leaderboard with 69.4, ahead of GPT-6.1 Sol at 64.6.
- Eight tabletop tasks test grasping, pouring, writing, tower building and measuring.
- Models are ~100x slower than humans, averaging 30 minutes per task versus 20 seconds.
- GPT-6 Astra costs $16.36 per task, 8x more than GPT-6.1 Sol at $1.93.
- Grasp success rates range 28-45%, with 70% of spend going to visual input context.
Artificial Analysis, which publishes comparative model evaluations, has introduced AA-Robotics v0.1, a benchmark that lets frontier language models control a Hiwonder NexArm. Each model receives a plain-text objective, documentation for an HTTP control API, feeds from four cameras, and an isolated container where it can write and run movement code.
The first results show that general-purpose coding agents can complete simple manipulation tasks without fine-tuning or robotics-specific controllers. Their reliability, latency, and operating costs remain far from production requirements: most runs take about 30 minutes per task, compared with a 20-second human baseline.
Astra leads; Sol costs far less
Each task receives a score of up to 100 points, with partial credit for progress. Artificial Analysis averages the eight task scores to produce the leaderboard result.
| Model | Score | Average time per task | Average cost per task |
|---|---|---|---|
| GPT-6 Astra | 69.4 | About 30 minutes | $16.36 |
| GPT-6.1 Sol | 64.6 | About 30 minutes | $1.93 |
| Gemini 4 Argon | 55.3 | 52 minutes | Not reported |
| Claude Opus 5.5 | 48.8 | About 30 minutes | Not reported |
| Claude Fable 5.1 | 46.3 | About 30 minutes | Not reported |
Because task scores include partial credit, leaderboard position can diverge from full-completion rates. GPT-6 Astra and Claude Fable 5.1 recorded the highest proportion of successful attempts at about 45%, while GPT-6.1 Sol had the lowest at 28% despite placing second overall.
Eight tests of control and recovery
The suite uses tabletop tasks to test spatial reasoning, contact handling, movement planning, error recovery, and the agent’s ability to recognize when it has finished:
- Press a red button.
- Place a crocodile toy inside a rectangle.
- Pour a drink from a bottle into a cup without spilling.
- Put four blocks into a box and replace the lid.
- Complete a shape-matching board from a reference image.
- Write
hello worldlegibly with a marker inside a black box. - Build the tallest freestanding foam-block tower possible.
- Measure a box’s length, width, and height within 2 mm.
All five models completed the button and crocodile tasks. The remaining exercises exposed failures in precision, physical reasoning, and completion detection. Box measurement was the hardest task, with most models receiving no credit.
Thirty-minute agents, 20-second humans
The benchmark’s human baseline averages 20 seconds per task. A 30-minute model run is about 90 times slower, while Gemini 4 Argon’s 52-minute average is roughly 156 times slower.
Planning accounts for a substantial share of that delay. Gemini 4 Argon waited an average of 21 minutes before taking its first action, nearly twice as long as any other model. On the button task, GPT-6 Astra finished in 2.2 minutes, while Claude Fable 5.1 spent about 10 minutes planning before making contact.
Among models with reported costs, GPT-6.1 Sol offers the stronger price-performance result. Its average task cost of $1.93 is about one-eighth of GPT-6 Astra’s $16.36, while its benchmark score trails by 4.8 points. Using the 20-second human baseline and the average US hourly wage, even Sol costs roughly 10 times as much as the corresponding human labor.
Artificial Analysis attributes about 70% of model spending to context, including the visual information needed to observe the scene repeatedly. That cost profile gives developers a concrete optimization target: fewer frames, tighter image crops, and structured state summaries could reduce context usage if they preserve enough detail for accurate control.
A sparse harness around a real arm
The benchmark limits task-specific scaffolding so that performance depends primarily on the model’s planning, coding, and visual reasoning. Each attempt follows the same basic loop:
- The model starts in an isolated container with the task prompt and arm API documentation.
- It observes the workspace through three table-mounted cameras and one wrist camera.
- It writes and executes code that sends movement commands through the HTTP API.
- It inspects new camera images, adjusts its plan, and continues acting.
- It decides when the task is complete and submits the result for human grading.
GPT models run through Codex, Claude models through Claude Code, and Gemini 4 Argon through Antigravity. Every system uses its high-effort setting. The attempts are zero-shot, meaning the models receive no worked examples, task-specific fine-tuning, or learned robot policy for these exercises.
Stopping is part of the task
Several failures came from poor completion detection rather than an inability to make useful progress. GPT-6.1 Sol built a passing tower after about 17 minutes, then continued adjusting it until the structure failed. Other agents accepted an off-center lid as complete or kept trying to grasp a bottle after it had tipped over and spilled.
| Failure mode | Observed behavior |
|---|---|
| Weak stopping criteria | The agent damages a valid result by continuing to manipulate it. |
| Spatial miscalibration | The gripper closes 1.3 cm above the intended object. |
| Poor contact prediction | A wrist press moves the entire tray instead of activating the target. |
| Unmodeled force | A stream from the bottle pushes the cup sideways during pouring. |
| Destructive probing | Pen-contact tests leave stray ink inside the required writing area. |
| Weak recovery | The agent repeats a failed grasp after the scene has materially changed. |
These errors show where camera-based planning remains unreliable: estimating contact, friction, momentum, alignment, and the consequences of another action. Production systems would need explicit verification, recovery logic, and safety limits around the model’s commands.
A generalist test for physical work
Many robotics benchmarks evaluate specialized systems. Diffusion policies generate action trajectories, vision-language-action models map images and instructions to controls, and imitation-learned controllers reproduce behavior from demonstrations. AA-Robotics instead measures how much control a general-purpose coding agent can assemble from API documentation and camera observations.
Artificial Analysis says frontier language models remain unsuitable for independent production use in robotics. The current results support that assessment: task completion is inconsistent, actions are slow, and repeated visual context is expensive. Future releases can show whether those constraints improve across model generations under a stable test setup.
The benchmark also expands Artificial Analysis’s work on long-horizon agent evaluation. Its related Knowledge Agent benchmark, developed with Google DeepMind, evaluates multimodal research tasks that require models to gather and synthesize information over extended runs.
Design implications for agent builders
- Budget for retries and verification. Full-success rates of 28% to 45% make unchecked physical actions unsuitable as dependable pipeline steps.
- Evaluate cost against marginal quality. GPT-6 Astra gains 4.8 points over GPT-6.1 Sol while costing about 8.5 times more per task.
- Reduce repeated visual context. Selective camera use, cropped views, and compact state representations may lower the dominant input cost.
- Define completion explicitly. A separate verifier or deterministic stopping rule could prevent agents from undoing successful work.
- Constrain recovery behavior. Agents need scene-change detection and bounded retry policies before repeated actions can be trusted around physical equipment.