UBC Builds a Synthetic Pipeline That Trains Video AI Without Real Annotations
A new pipeline turns single images into fully annotated multimodal videos, then trains models with structured Q&A instead of captions.
Takeaways
- Unified pipeline generates video, captions, masks, counts, and VQA pairs from a single image, using Wan 2.2, SAM2, and MUG-VOS.
- VQA-based fine-tuning outperforms caption-based supervision on counting and video question answering benchmarks.
- InternVideo2.5 fine-tuned on synthetic data cuts MSCOCO counting MAE from 5.46 to 3.18.
- Segmentation mIoU improves by +5.28 points on MSCOCO after fine-tuning on 5K synthetic videos.
- Scaling from 2K to 5K synthetic videos consistently improves segmentation, suggesting further gains are available.
- No public code or model weights released with the paper; evaluated at 5K-video scale.
Training multimodal models to understand video is a data problem before it is a modeling problem. Every second of footage needs frame-level labels, tracked objects, and consistent supervision across time, and those annotations cost real money. A team from the University of British Columbia and the Vector Institute wants to sidestep that bottleneck with a synthetic data pipeline that starts from a single image and produces text, video, segmentation masks, and optional audio, all automatically aligned.
The paper, titled All in One, argues that models fine-tuned almost entirely on this generated data can match or beat models trained on real annotated video across counting, question answering, and segmentation benchmarks.
Why annotated video is such a chokepoint
Collecting and labeling multimodal video in the real world is slow, expensive, and limited in diversity. Static images already require box and mask labels; video multiplies that by every frame and adds temporal consistency, object tracking, and event structure. The result is a supply problem: datasets are uneven across tasks, biased toward whatever was easy to collect, and rarely cover the compositional edge cases that trip up multimodal LLMs.
Synthetic data has been a workaround for years, though earlier approaches tended to specialize. You would find one pipeline for segmentation masks, another for captions, another for QA pairs. What is new here is that a single generation process emits all of them, tied to the same underlying scene.