A robot arm, four cameras, and an LLM

General-purpose frontier models can now use camera images, Python, and an HTTP API to manipulate a real robot arm without task-specific examples. In AA-Robotics v0.1, Artificial Analysis measures how well vision-capable language models can understand a tabletop scene, plan movements, and control physical hardware. The research preview provides an early baseline for embodied reasoning with standard coding-agent tools.

AA-Robotics v0.1 covers eight tasks performed by a Hiwonder NexArm with five articulated joints and a gripper. Each model receives a simple objective and the arm’s product name, but little information about its kinematics, the relationship between joint movements and the gripper’s position.

From camera frames to joint commands

Each model runs inside an isolated container with instructions for reaching the arm, sending commands, and submitting a completed attempt. It observes the workspace through four cameras, including three fixed tabletop views and one wrist-mounted view, then writes and executes Python to issue joint commands through an HTTP API.

• Hardware: A Hiwonder NexArm with five joints and a gripper.
• Visual input: Three external camera feeds and one wrist camera.
• Control interface: Python code sending joint positions through HTTP.
• Task setup: A short objective with no worked examples or task-specific training.

After a model submits an attempt, the benchmark grades the final state against task-specific pass criteria. A complete pass earns 100 points, while incomplete attempts can receive partial credit. Here, zero-shot means the model receives no demonstrations for the task, although it may draw on information acquired during pretraining.

The first eight tasks test three capability groups:

• Control and dexterity: Grasping, moving, and placing objects accurately.
• Spatial reasoning: Predicting how objects interact during actions such as stacking.
• Adaptability: Recovering when the environment changes during an attempt.

Two models define the cost frontier

GPT-6 Astra leads the initial leaderboard with a score of 69, followed by GPT-6.1 Sol at 65. Gemini 4 Argon and the two Claude models trail the GPT systems in the eight-task preview.

The published cost-to-score comparison places only Astra and Sol on the Pareto frontier, where no competing model is both cheaper and higher-scoring. Astra’s four-point advantage costs an additional $14.43 per task, making it about 8.5 times as expensive as Sol. Exact per-task costs for the other models are absent from the summary figures.

Grasps, geometry, and false finishes

Across the reported runs, failures cluster around physical precision, spatial planning, and the model’s ability to judge its own progress:

1. Missed grasps and placements: Objects are dropped, pushed aside, or moved beyond the arm’s reach.
2. Faulty spatial plans: Models choose the wrong action order or knock over structures they have already assembled.
3. Incorrect completion judgments: Models continue manipulating a scene after reaching a passing state or stop before the task is complete.

Artificial Analysis reports that the strongest models take about 30 minutes per task on average. Under the benchmark’s comparison, that is roughly 100 times slower and 10 times more expensive than a person performing the same tabletop work. The latency alone excludes many production workflows with short cycle times.

Eight tasks set tight boundaries

Results from eight tasks on one arm and one tabletop environment measure a narrow form of physical competence. Several limits affect how the scores should be interpreted:

• Limited task coverage: The preview does not establish performance on long-horizon work, deformable objects, cluttered environments, or mobile robots.
• Hardware specificity: Success with the NexArm does not guarantee transfer to robots with different joints, grippers, payloads, or control interfaces.
• End-state grading: A final-state score may omit inefficient motion, collisions, and unsafe intermediate actions unless the pass criteria explicitly capture them.
• Ranking uncertainty: Without confidence intervals or repeat-trial counts in the reported figures, small score differences offer limited evidence about reliability.

A coding agent becomes the controller

AA-Robotics routes control through a general coding-agent harness rather than a specialized vision-language-action policy trained to map images and instructions directly to robot movements. The model interprets camera frames, generates control code, observes the result, and adjusts its next action.

That loop shows that a general multimodal model can connect perception, planning, and actuation on constrained tabletop tasks without task-specific fine-tuning or a teleoperation dataset. Each observation, inference, code-generation, and correction cycle also adds latency, making faster inference and more accurate motion planning the clearest routes to shorter runs.

Independent projects such as Robocurve are developing related open-source tools and benchmarks for tracking robotics capabilities. Repeated public evaluations can reveal whether gains in model speed and reasoning translate into faster, cheaper, and more reliable physical control.

Best used as a measurement tool

AA-Robotics v0.1 gives developers and researchers a compact environment for testing how general-purpose agents behave when their outputs move physical hardware. Its immediate uses include:

• Evaluating whether an agent harness can connect visual perception, planning, code generation, and actuation.
• Comparing frontier models on embodied reasoning alongside text and software benchmarks.
• Measuring changes in task cost, completion time, reliability, and recovery behavior across model releases.
• Cataloging failure modes before experimenting with broader tasks or more expensive hardware.

Production manipulation systems still require calibrated perception, motion planning, collision checking, deterministic limits, emergency stops, monitoring, and extensive validation. AA-Robotics offers a reproducible baseline for tracking how much of that workflow general-purpose models can eventually handle.