Skip to main content
Back

UIUC's Brick-Composer Teaches AI to Build LEGO Structures, Tripling Assembly Accuracy

Brick-Composer teaches multimodal LLMs to assemble LEGO-style objects step by step, boosting success rates from near zero to 15% with a new benchmark and three-signal training framework.

  • New benchmark: BC-Bench is the first benchmark for evaluating MLLMs on fine-grained brick selection and 3D pose estimation in assembly tasks.
  • Baseline models fail: State-of-the-art MLLMs achieve under 1% strict step-level assembly success out of the box, exposing a major gap in spatial reasoning.
  • Three-signal training: Brick-Composer trains on Human Design Sparks, World Feedback (visual/physical consequences), and Synthetic Experience to scale beyond 102 real objects.
  • 3x+ improvement: Brick selection accuracy triples and strict step-level success rises from under 1% to ~15% after training.
  • Qwen-3-8B reaches 42%: The fine-tuned 8B model correctly completes up to 42% of consecutive assembly steps for a full object.
  • Open source: Code and benchmark available at github.com/Lumos-Jiateng/Brick-Composer.

Teaching an AI to build something from physical parts sounds deceptively simple. You show it a design, it picks the right piece, places it correctly, and repeats. But this task turns out to expose a deep gap in what today's best multimodal large language models (MLLMs) can actually do. A new paper from researchers at UIUC, Stevens Institute of Technology, and Northwestern University takes this challenge head-on with Brick-Composer, a learning framework designed to give MLLMs real assembly skills using LEGO-style bricks.

The team's ambition is AI agents that can read arbitrary designs and construct real-world objects from reusable building blocks, and they start by asking whether MLLMs already have the visual grounding and spatial reasoning needed for brick assembly. The short answer: not even close. But the gap is now measurable, and Brick-Composer is a concrete first step toward closing it.

Why bricks are a surprisingly hard test

A capable assembly agent must recognize building blocks from fine-grained geometric features and functional affordances, infer each brick's target pose within the correct assembly sequence, execute reliable actions in the physical world, and continuously monitor the long-horizon construction process while remaining able to recover from errors along the way. That is a lot to ask of a model that was primarily trained on text and images.

The paper formulates brick assembly as a sequential decision-making problem, where each step involves two subtasks: brick selection, identifying the target brick from candidate components, and brick pose estimation, predicting where and how the selected brick should be placed. Both subtasks must succeed at every step for the assembly to be correct. One wrong pick or a slightly off rotation cascades into a broken structure.

This is also a domain where prior benchmarks fell short. LEGO-Puzzles, a related benchmark, assesses multi-step spatial reasoning in MLLMs using LEGO assembly tasks across three levels: spatial understanding, single-step reasoning, and multi-step sequential reasoning. But it focuses on visual question answering, not on actually predicting the physical placement of a specific part. Brick-Composer is the first work to address the lack of systematic evaluation of MLLMs for fine-grained brick selection and single-step pose estimation in realistic assembly settings.