Harvard's ComSim Multiplies Robot Training Data 20x Without Extra Teleoperation
A hybrid pipeline turns a handful of real robot demos into hundreds of photorealistic training videos, boosting Diffusion Policy success rates from 3/30 to 17/30 out-of-distribution.
ProTakeaways
- ComSim combines classical and neural simulators to generate photorealistic robot training videos with accurate action labels.
- 10 real demos plus 200 synthesized demos beat 20 real demos on eight manipulation tasks.
- Compositional Dynamic Video Generation composes Control Dynamics and Visual Dynamics scores in a diffusion transformer.
- Diffusion Policy jumped from 3/30 to 17/30 out-of-distribution on Move Playing-Card Away using ComSim data.
- New-object generalization on Shake Bottle went from 0/30 (20 Real baseline) to 15/30 with pseudo-real data.
- Code and pipeline released at github.com/faceong/ComSim; built on Stable Diffusion 1.5 and RoboTwin.
Robot learning has an awkward math problem. Policies get more capable every quarter, but the data that trains them still comes from humans painstakingly teleoperating arms one trajectory at a time. A new paper from a team spanning CUHK-Shenzhen, Sun Yat-sen University, Oxford, and Harvard proposes a way out that avoids picking a side in the classical-versus-neural simulator debate. It runs both, and stitches them together.
The system is called ComSim (Compositional Simulation). Using a closed-loop real-sim-real data augmentation pipeline that leverages a small amount of real-world data, the authors train a neural simulator to transform classical simulation videos into real-world representations, improving policy models trained in real-world environments. In their experiments, ten real demonstrations plus 200 synthesized ones beat twenty real demonstrations by a wide margin, and the gap widens further when the robot faces new spatial layouts or unseen objects.
Two flawed sources of robot data
Robot training data today comes from three main sources, as the paper frames it: direct human collection, which yields high-quality samples but cannot scale; classical simulators, which generate large datasets but suffer from appearance and physics gaps to reality; and neural simulators trained on real data, which reduce these gaps but struggle with action-conditioned video generation, leading to weak action-video consistency.

Classical simulators like MuJoCo, Isaac, and SAPIEN are precise about actions but look wrong: lighting is off, textures are synthetic, and contact physics is approximate. Neural simulators built on video diffusion models look right but hallucinate. When you condition a video model on an action sequence, the pixels it produces do not always reflect what that action would actually do. Neither source alone gives you the (video, action) pairs that policy learning needs.