CMU's Pixel-UMI Trains Mobile Robots From Handheld Demos at 87% Success
A CMU pipeline turns handheld human demos into whole-body mobile robot policies that avoid walls, reach into fridges, and fold boxes at 87% task progress.
- CMU's Pixel-UMI learns whole-body mobile manipulation from handheld human demos, hitting 87% average task progress.
- Environment-Aware Retargeting uses reconstructed scene geometry to generate collision-free robot postures from human head and gripper trajectories.
- Retargeting-Error-Aware Masking hides mismatched camera views during training so policies rely on observations the robot can reproduce.
- Delay-calibrated action assignment accounts for per-joint controller lag when converting trajectories into training action labels.
- Two DAgger rounds with ~60 corrected rollouts enable contact-rich bimanual tasks like box folding and jacket hanging.
- Paper on arXiv; code promised but not yet released.
Carnegie Mellon researchers have introduced Pixel-UMI, a system that converts demonstrations recorded with a sensorized handheld gripper into training data for a wheeled robot with a torso and two arms. The accompanying paper reports 87.0% average task progress across five mobile-manipulation tasks. The metric rewards completed stages within a task, including runs that stop before full completion.
Handheld recording lets operators collect demonstrations without occupying a robot or building a teleoperation station for every worker. Pixel-UMI addresses the difficult transfer step: fitting human head and hand trajectories to a robot whose body, joints, cameras, and control delays differ from the demonstrator’s.
Human motion meets robot geometry
The project extends the Universal Manipulation Interface, or UMI, introduced by Chi and collaborators. UMI records demonstrations through handheld grippers and represents actions as relative trajectories, while matching policy timing to the target hardware’s latency. That design has worked well for tabletop, single-arm manipulation.
Mobile manipulation adds a wheeled base, torso, second arm, head camera, and more opportunities for collision. When a person reaches into a refrigerator, their shoulders, elbows, and head follow paths that a specific robot cannot reproduce joint for joint. Directly mapping the recorded head and gripper poses can drive the robot into doors, counters, boxes, or chairs.
Pose errors also change the camera views available during deployment. A policy trained on images from the demonstrator may receive substantially different wrist or head images once the same trajectory has been fitted to the robot, weakening the connection between visual observations and learned actions.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
