Ai2's ToyBlocks Gives Robotics Teams 85K Hours of Controlled Training Data

The Allen Institute for AI has released ToyBlocks, an open simulation framework in which a robot arm manipulates wooden blocks on a tabletop. Its narrow setting gives robotics teams reproducible control over task composition, training data, object placement, friction, and other variables that are difficult to isolate on physical hardware.

The full experimental loop

• Simulation assets for the robot, blocks, and tabletop.
• Procedural generators for tasks and demonstrations.
• Two trained models with training and evaluation code.
• Validation sets for measuring policy behavior.
• Apache 2.0 licensing across the release.

The initial dataset contains 85,336 long-horizon demonstrations, each chaining roughly six to 12 skill executions. Together, they represent nearly 1,600 hours of robot activity and 144 million frames.

A tabletop built for controlled answers

Robot-learning datasets are commonly collected through teleoperation, with a person guiding the robot through each trajectory. That process is slow, expensive, and difficult to vary systematically, which complicates studies of how task selection, data mixtures, and environment changes affect generalization.

ToyBlocks is the first entry in Ai2’s FoundationWorlds series. Each simulated world targets a specific robotics domain and combines procedural task generation, automated demonstrations, and behavioral metrics. The tabletop setting lets ToyBlocks pursue realistic physics and appearance, scalable data generation, multi-step reasoning, dexterous control, and broad coverage within a bounded domain.

Demonstrations at GPU speed

Inside Nvidia’s Isaac Lab, ToyBlocks uses reinforcement learning and motion planning to train short-horizon skills such as pushing, flipping, and stacking. A task-and-motion planner composes those skills into longer demonstrations. During generation, the system can use exact simulator state, including each block’s position and orientation, so it can create trajectories without a human operator recording every movement.

On a single GPU, the framework can generate and execute hundreds of plans in parallel at up to 24 times real-time speed. It records datasets in the LeRobot format for use in existing vision-language-action pipelines, which train robots to convert visual observations and language instructions into controls.

Held-out layouts expose weak generalization

In Ai2’s initial tests, ManiFlow, pi0.5, and MolmoAct 2 performed poorly on spatial configurations omitted from training. The held-out cases included new regions of the tabletop and stack heights of three or four when the training data contained only heights of two and five. Large-scale pretraining did not produce reliable spatial generalization in these experiments.

To diagnose failures across longer tasks, Ai2 pairs Molmo 2 as a high-level planner with ManiFlow as the low-level controller. The system handles tasks such as arranging lettered blocks to spell “Stack Ai2.” ToyBlocks can replace either layer with an oracle component that has access to ground-truth answers, allowing researchers to attribute errors to the planner, controller, or their interface.

An agentic experiment connected Claude Opus 5.5 to ToyBlocks skills through the Model Context Protocol. The agent solved or correctly rejected as infeasible 63% of search-and-spell tasks. The reported errors were dominated by confusion between similar-looking letters, at 13%, and incorrect block orientation, at 15%.

Skill descriptions also affected performance. Default descriptions produced a 24% success rate, while descriptions written by a human expert reached 36%. Allowing the agent to run experiments and write descriptions from its observations raised success to 63%.

Simulation stays close to hardware

ManiFlow transferred from ToyBlocks to a Franka FR3 arm without real-world fine-tuning. Its single-skill success rate was 80.0% in simulation and 81.1% on the physical robot. The hierarchical Molmo 2 and ManiFlow system succeeded in 55.5% of simulated episodes and 50.0% of real-world episodes.

• System identification: Ai2 calibrated joint friction and armature parameters to match the physical robot.
• Domain randomization: Data generation varied physical and visual properties so the policy would encounter a wider range of conditions.
• Stereo depth: Paired RGB images produced clean depth maps at inference time, avoiding dependence on a hardware depth sensor.

Questions ToyBlocks can test

• Whether depth input improves control success over RGB alone.
• Whether a policy can stack four blocks without four-block demonstrations.
• How much data a vision-language-action model needs before spatial generalization improves.
• How action-conditioned world models, which predict future states from actions, compare with ground-truth simulator state.
• Which training mixtures produce a particular policy failure.

A browser interface can connect a frontier model to the simulator and control the arm through natural-language instructions, giving developers a way to inspect planning behavior without modifying the training stack.

More skills and more complex hands

Ai2 says a second release is planned within weeks, with a larger dataset, additional skills, broader benchmarks, and integrated reinforcement-learning tools absent from the current package. Longer-term work includes bimanual manipulation, dexterous hands with force and touch sensors, and tasks involving unstructured piles of blocks.