Peking University's WM-DAgger Trains Robots to 93.3% Success With 5 Demos
A new framework uses world models to synthesize robot recovery data automatically, hitting 93.3% success on soft bag pushing from just five demonstrations.
ProTakeaways
- WM-DAgger uses world models to auto-generate robot recovery data, replacing the human-in-the-loop bottleneck of classical DAgger
- Hits 93.3% success on soft bag pushing from just 5 demonstrations, versus 26.7% for standard behavioral cloning
- Built on Cosmos-Predict 2.5 (2B) with a new Action2Image module for dense geometric conditioning
- Two modules make it work: constrained corrective action synthesis and DINOv2-based hallucination filtering
- Beats diffusion baseline DMD across four tasks including pick-and-place, ballot insertion, and towel folding
- Code released at github.com/czs12354-xxdbd/WM-Dagger; dexterous multi-finger hands flagged as future work
Imitation learning has a nagging problem: robots trained by copying human demos tend to drift off course, and once they enter a state the demonstrator never showed them, small errors snowball into failure. A team from Peking University, JD Logistics, HKUST-Guangzhou, Rutgers, and UT Dallas is proposing a fix called WM-DAgger, which uses a generative world model to fabricate the recovery trajectories that a human trainer would otherwise collect by hand.
On a soft bag pushing task, their policy hits a 93.3% success rate with only five demonstrations, versus 26.7% for standard behavioral cloning on the same data budget. The world model does the tedious work of generating out-of-distribution corrections that DAgger traditionally requires a human operator to provide live.
Why behavioral cloning keeps falling over
Imitation learning suffers from compounding errors: minor policy inaccuracies push robots into unseen out-of-distribution states, where the policy generates even bigger errors, leading to eventual failure. The classical remedy is DAgger, where an expert stays in the loop and demonstrates how to recover whenever the robot wanders off. Continuous human involvement severely limits scalability.
Earlier attempts to automate this used diffusion models to synthesize recovery images, but those methods often prioritize visual synthesis over physical awareness, and they generate single frames rather than continuous dynamics. That is a real limitation for tasks involving deformable objects like cloth or soft bags, where the physics matter.