ByteDance's OmniShow Generates Videos of People Holding Products With Voice and Motion
Researchers from CUHK and ByteDance introduce OmniShow, a 12B video model that unifies text, reference images, audio, and pose into one framework.
ProTakeaways
- Researchers from CUHK and ByteDance introduce OmniShow, unifying text, reference images, audio, and pose for video generation.
- The 12.3B model, built on Waver 1.0, adds only 0.3B extra parameters through targeted audio attention placement.
- Three innovations: Unified Channel-wise Conditioning, Gated Local-Context Attention, Decoupled-Then-Joint Training with model merging.
- Merging separate R2V and A2V specialists produces emergent reference-plus-audio capabilities before any joint training.
- New HOIVG-Bench with 135 samples evaluates text alignment, reference consistency, audio-sync, pose accuracy, and video quality.
- Beats HuMo-17B, Phantom-14B, VACE on multiple metrics; project page live but no code released yet.
Researchers from The Chinese University of Hong Kong and ByteDance have released a paper introducing OmniShow, a video generation framework that accepts text prompts, reference images, audio, and pose sequences simultaneously and produces coherent videos of a person interacting with an object. The work formalizes a new task the authors call Human-Object Interaction Video Generation (HOIVG) and argues that no existing open model handles all four modalities at once.
The setup targets a concrete pain point in content creation: making a specific person hold a specific product, move in a specific way, and speak specific words, all in one generation pass. Current systems force you to pick two or three conditions at most.
Why the existing toolbox falls short
The paper walks through why this task is hard. Reference-to-video models like Phantom preserve subject identity but produce silent characters. Audio-driven avatar models like Hallo3 or OmniAvatar sync lips to speech but only accept a single starting frame, not separate human and object references. Pose-guided systems like AnchorCrafter and DreamActor-H1 lock down motion but ignore audio entirely. Chaining these tools together, the authors show, causes visible artifacts, especially around the mouth when a lip-sync module is bolted on as post-processing.
Beyond the modeling gap, two other blockers stand out: paired training data with all four conditions valid at once is scarce, and no standardized benchmark existed for evaluating this specific task.
