Skip to main content
Back

Brown University's GenHSI Makes AI Place People on Furniture Without Training

A training-free pipeline from Brown University generates long videos of a character interacting with a scene using 3D-aware keyframes and off-the-shelf diffusion models.

Paper, Full Access
  • Brown University's GenHSI generates long human-scene interaction videos with no training required.
  • Three-stage pipeline: script writing with a VLM, 3D keyframe previsualization, animation via off-the-shelf diffusion.
  • Inpainting humans in the object's canonical view hits 93% success versus 7% at 90 degrees rotation.
  • Beats Kling AI 1.6 Elements on subject consistency (0.985 vs 0.961) and imaging quality.
  • Uses Trellis, OrientAnything, SMPL-X, HybrIK-X, LHM Gaussian avatars, and Kling for interpolation.
  • Code available on GitHub; paper accepted at WACV 2026.

Video diffusion models can produce eye-popping clips, but ask one to render a specific person walking across a specific room, sitting on the left chair, then lying on the couch, and things fall apart. Characters slide through furniture, identities morph mid-clip, and the model happily invents a table that was never in your reference image. A team from Brown University has proposed a workaround called GenHSI that fixes many of these failures without training or fine-tuning anything.

GenHSI is a training-free framework for generating controllable, long human-scene interaction (HSI) videos from minimal user inputs: an image of a scene, images of a person, and a natural language description of the desired interaction sequence. The paper was accepted at WACV 2026, and the authors have released code on GitHub.

Why current video models trip over furniture

The team pinpoints three specific failure modes in modern video diffusion models. Because dynamics are only implicitly formalized in visual generative models, outputs often show unrealistic physical behavior, especially when several dynamics happen at once. Without explicit 3D modeling, VDMs frequently fail to ground text instructions in accurate spatial reasoning, producing artifacts such as characters heading in wrong directions or hallucinated 3D affordances during interaction. Keeping a person looking like the same person over a long clip is also hard, even with expensive fine-tuning.

Affordance, in this context, means the surfaces and orientations that support a given action: a chair affords sitting, a couch affords lying. When a diffusion model has no explicit 3D grounding, it guesses where those surfaces are, and the guesses are often wrong.