Skip to main content
Back

Cornell's X-Diffusion Trains Robots on Human Videos With 16% Better Success

Cornell researchers use Ambient Diffusion to treat human videos as noisy robot data, boosting manipulation success rates by 16 percent across five real-world tasks.

  • X-Diffusion trains Diffusion Policies on human videos by treating human actions as noisy robot actions.
  • A classifier finds the earliest noise level where human and robot actions become indistinguishable.
  • Human demos supervise the policy only at diffusion timesteps beyond that threshold, avoiding infeasible moves.
  • Achieves 16 percent higher average success rate over naive co-training across five real Franka tasks.
  • Outperforms even manually filtered human demonstrations, extracting signal from infeasible trajectories.
  • Code on GitHub and data on Hugging Face.

Robot learning has a data problem. Teleoperating a robot arm to collect demonstrations is slow, expensive, and requires expert operators, while videos of humans doing the same tasks are essentially free. The catch is that a human hand can slide fingers under a plate to pick it up, whereas a robot with a two-finger gripper physically cannot. Train a policy on mixed human and robot data and you often get one that confidently attempts moves the hardware cannot execute.

A team from Cornell has released X-Diffusion, a training recipe that borrows an idea from image generation to solve this. The trick is to treat human demonstrations as noisy versions of robot demonstrations, and only let them supervise the policy at diffusion timesteps where the embodiment gap has been drowned out by noise. Across five real-world manipulation tasks, X-Diffusion beats naive co-training and even manual data filtering by an average of 16 percentage points in success rate.

Why mixing human and robot data usually backfires

The standard approach to using human videos is retargeting: run a 3D hand-pose estimator, map the human hand trajectory to end-effector poses, and co-train the robot policy on the combined dataset. The paper points out that these mappings unify the representation of actions but not their physical realizability. Humans routinely use side grasps, finger-sliding, and contact strategies that a parallel-jaw gripper cannot reproduce.

When a Diffusion Policy is trained on both distributions with equal weight, it learns to reproduce whatever style dominates the data, including moves that are dynamically infeasible on the robot. The authors show this failure mode explicitly: on Push Plate and Pan On Plate, baselines that co-train on all human data end up attempting side grasps and failing repeatedly. In their filtering study, roughly half of raw human demonstrations turned out to be kinematically or dynamically infeasible when replayed through inverse kinematics on the real robot.