Skip to main content
Back

Max Planck's GraG Reconstructs Hand-Object Grasps 6.4x Faster From Video

Researchers from Max Planck Institute, Google, and Bielefeld University cut monocular hand-object reconstruction from hours to 30 minutes.

Pro

Paper, Full Access
  • Max Planck, Google, and Bielefeld researchers released GraG, monocular hand-object 3D reconstruction.
  • Runtime drops from 3-10 hours to 30 minutes on long video sequences.
  • 6.4x faster than prior work with 13.4% better object reconstruction quality.
  • Reduces hand per-joint position error by over 65 percent.
  • Combines SAM3D generative prior with a compact Sum-of-Gaussians tracker.
  • Code available at github.com/aidilayce/GraG with interactive viewer online.

Reconstructing the 3D geometry of a hand grabbing, twisting, or manipulating an object from a single video has been one of computer vision's slow, painful problems. A team from the Max Planck Institute for Informatics, Google, and Bielefeld University just released Grasp in Gaussians (GraG), a method that produces temporally stable hand-object reconstructions from monocular video in roughly the time it takes to grab lunch, rather than an entire afternoon.

GraG reconstructs temporally coherent hand-object interactions on long sequences 6.4x faster than prior work while improving object reconstruction by 13.4% and reducing hand per-joint position error by over 65%. In wall-clock terms, runtime drops from 3-10 hours to 30 minutes on long sequences.

Why hand-object video breaks reconstruction

Reconstructing hands manipulating objects from one camera view means fighting several problems at once. The hand occludes the object, the object occludes the hand, both are moving, and neither has a known template if you want a category-agnostic system. Recent Gaussian-splatting pipelines like BIGS handle occlusion by leveraging amodal completion to infer the complete structure of partially obscured regions and integrating temporal context across video sequences to incrementally refine and stabilize reconstructions. The results are good, but the optimization is heavy and slow.

The GraG authors point out that image-to-3D foundation models and classical trackers each solve half the problem. Recent image-to-3D models can produce plausible object geometry from a single view, but naively re-running such models frame-by-frame yields jittery poses and inaccurate shape under occlusion. Classical model-based human and object tracking formulations are fast and stable, but have historically required carefully engineered templates or multi-view capture, which cannot directly generalize to hand-object reconstruction. GraG stitches the two together.