NVIDIA's ViPE Turns Ordinary Video Into 3D Geometry With 96M Annotated Frames

NVIDIA's Toronto AI Lab has released ViPE, a video processing engine that pulls calibrated 3D geometry out of ordinary video clips. Point it at a selfie, a dashcam recording, a cinematic pan, or a 360-degree panorama, and it returns per-frame camera intrinsics, camera pose, and dense near-metric depth maps. It ships under Apache 2.0, installs from PyPI as nvidia-vipe, and comes with a companion dataset of roughly 96 million annotated frames.

The messy problem it cleans up

Training spatial AI, world models, and 4D reconstruction systems requires video labeled with accurate camera trajectories and depth. Getting consistent, precise 3D annotations from in-the-wild footage remains a stubborn problem, and existing pipelines break the moment a scene contains moving objects, an unknown lens, or a fisheye field of view. Classical structure-from-motion tools assume static scenes and calibrated cameras. Recent learned methods either struggle with scale drift or fall over on long, dynamic clips.

ViPE estimates camera intrinsics, camera motion, and dense near-metric depth from unconstrained raw video, and handles dynamic selfies, cinematic shots, and dashcams across pinhole, wide-angle, and 360-degree panorama models. That last part matters, because most existing pose estimators quietly assume a pinhole camera.