Tsinghua's ShapeGen Teaches Robots to Handle Any Object From Just 5 Demos
Tsinghua researchers built a simulator-free pipeline that warps 3D shapes into functional twins, generating diverse manipulation demos from a single teleoperated example.
Takeaways
- ShapeGen generates shape-diverse manipulation demos from one teleoperated source, no simulator required.
- Neural spatial warpings trained purely on SDF geometry map functionally corresponding points across in-category shapes.
- Boosts ManiFlow policy success from 1/20 to 9/20 on unseen mugs for hang_mug task.
- Requires roughly one minute of human keypoint annotation per source demo, then fully automated.
- Adding new shapes to the library is O(1) via composition through a common template shape.
- Code coming soon on the project page; no articulated or deformable object support yet.
Teaching a robot to hang a mug is easy. Teaching it to hang any mug, regardless of handle shape, wall thickness, or rim curvature, is where imitation learning falls apart. A new paper from Tsinghua University introduces ShapeGen, a data generation framework that tackles this by synthetically swapping the manipulated object in a single demonstration with a library of geometrically diverse shapes, while preserving the physics and function of the interaction.
Given a source demonstration with minimal human annotation, ShapeGen automatically substitutes task-relevant objects with objects of the same category but different shapes in a physically plausible and functionally correct way, yielding a large corpus of novel training data. No simulator required.
Why category-level generalization is the wall
Real-world manipulation policies fail on objects they were not trained on because the feasible action trajectory depends on the target's exact geometry. The usual fix is to collect more data across more object instances, but that scales poorly. Prior data generation work has explored several axes of diversity while largely dodging shape itself.
- Texture and lighting augmentation leaves geometry frozen.
- Spatial configuration methods reshuffle where objects sit, not what they look like.
- Methods like RoboSplat and ReBot augment shape by replaying a task trajectory on a novel object, but are confined to simple pick-and-place tasks since they do not process affordance-relevant information.
- CP-Gen models the skill as keypoint-trajectory constraints, but works solely in simulation and only supports axis-aligned scaling, introducing far less diversity than the complex, non-linear deformations seen in real-world scenarios.