Imperial College's HO-Flow Generates Realistic 3D Hand Motions With 98% Physical Accuracy
A new framework pairs an interaction-aware VAE with masked flow matching to synthesize realistic 3D hand-object motions from text, cutting interpenetration and boosting diversity.
ProTakeaways
- HO-Flow generates 3D hand-object interaction motions from text and object point clouds, accepted at ECCV 2026.
- Interaction-aware VAE projects object point clouds into all 16 MANO joint frames for contact-rich latents.
- Masked auto-regressive flow matching replaces discrete tokens or frame-wise diffusion for continuous, temporally coherent generation.
- Pre-training on GraspXL's five million synthetic grasps drives large gains in generalization to unseen objects.
- Beats LatentHOI on GRAB, OakInk, and DexYCB in physical plausibility, interpenetration, and diversity; see the paper.
- Inference takes about 7 seconds per sequence on an H100; code is not yet released.
Generating a 3D hand that convincingly picks up, moves, and manipulates an object from a text prompt sounds simple, but it is one of the messier problems in computer vision. Fingers phase through geometry, contacts float in mid-air, and motion sequences jitter across frames. A new paper from Imperial College London and Inria, HO-Flow, tackles this with a two-stage system that pairs an interaction-aware autoencoder with a masked flow-matching model, posting state-of-the-art numbers on three standard benchmarks.

Why hand-object motion is such a hard target
Realistic 3D hand-object interactions have obvious uses in character animation, virtual reality, and robotic manipulation, but extending static grasp synthesis into full temporal sequences introduces new failure modes. The central headache is preventing physical inconsistencies, particularly interpenetration artifacts where the hand geometry passes through the manipulated object during motion.
Two families of prior approaches dominate the field, and both have weaknesses:
- Direct diffusion on raw poses: Early methods apply diffusion models directly to sequences of raw hand and object poses. The high degree of freedom of articulated hand models is expensive to sample from, and treating poses as unconstrained independent variables often fails to faithfully capture the underlying interaction distribution, producing floating hands or implausible contacts.
- Latent diffusion, frame by frame: More recent work embeds hand-object poses into a compact latent space, but existing latent representations are typically learned frame-independently, which weakens temporal coherence and does not encode fine-grained, contact-aware cues.
- Discrete auto-regressive tokens