Skip to main content
Back

SJTU's LIDEA Lets Human YouTube Videos Replace 80% of Robot Training Data

A new framework lets robots learn from human videos by aligning 2D features and 3D geometry, replacing up to 80% of costly teleoperation demos.

Pro

Paper, Full Access
  • LIDEA closes the human-robot embodiment gap using feature distillation plus 3D geometry alignment. Paper
  • Human video demonstrations can replace up to 80% of robot teleoperation data without hurting task success.
  • Introduces HPP-5M, a dataset of ~5M paired human and pseudo-robot frames from DexYCB, TACO, OakInk, OakInk2.
  • Uses a filter-and-fill trick: remove hand/arm from point clouds, insert a canonical virtual gripper for consistent 3D observations.
  • On Prepare Bread with 8 robot + 48 human demos, LIDEA hits 80/53/46% versus 33/0/0% for pixel-editing baselines.
  • Code and dataset promised on the project page; policy built on RISE-2 with DINOv3 backbone.

Training robot policies is stuck on a supply problem. Teleoperated demonstrations are slow and expensive to collect, while YouTube overflows with footage of humans manipulating objects. The catch: a human hand and a robot gripper look and move so differently that policies trained on one rarely transfer cleanly to the other. A team from Shanghai Jiao Tong University has released LIDEA, an imitation learning framework that closes this embodiment gap on two fronts at once and shows that human data can substitute up to 80% of robot demonstrations without hurting task success.

Why pixel editing keeps failing

Prior work has tried to bridge human and robot demos by literally rewriting the video. These cross-embodiment transfer strategies typically rely on visual editing, and they often introduce artifacts because of intrinsic discrepancies in appearance and 3D geometry. When you inpaint a human hand out of a frame and paste in a robot arm, you get plausible-looking RGB alongside broken depth, inconsistent lighting, and geometry that a 3D policy cannot trust.

Other lines of work either co-train on human and robot tokens (which suffers from kinematic mismatch, since a five-fingered hand does not map cleanly onto a parallel gripper) or extract object-centric affordances (which depend on brittle state estimators that struggle with deformable or articulated objects). LIDEA sidesteps all three failure modes by aligning representations rather than pixels.

Two alignments, one policy

The framework has two complementary components that both feed into a diffusion-based 3D policy. The authors use RISE-2 as a fixed baseline so gains can be attributed to data alignment rather than architecture.