Skip to main content
Back

Apple's Depth Pro 2 Turns Any Video Into Precise 3D Depth Maps

Apple's new depth model handles single images, video, and live feeds at native resolution, fixing the sharpness-versus-jitter tradeoff plaguing monocular depth.

  • Apple released Depth Pro 2, a monocular metric depth model for images, video, and live feeds.
  • Four sizes released, from 24M (S) to 872M (H+) parameters, plus a shared 23.5M focal length estimator.
  • Architecture uses a multi-view transformer alternating within-frame and cross-frame attention, with a DPT decoder.
  • Claims a new Pareto frontier on sharpness (Boundary F1) and temporal consistency (jitter) on Sintel and Spring.
  • Install via pip from the GitHub repo; runs in bfloat16 on GPU.
  • Weights are research-only under the Apple ML Research Model license, no commercial use permitted.

Apple’s machine learning research team has released Depth Pro 2, a four-checkpoint family for monocular depth estimation. Each model converts RGB input from one camera into a per-pixel depth map measured in meters, whether the source is a photo, video, or live stream. Apple says the models produce depth at the input resolution, preserve fine object boundaries, and maintain stable geometry across frames. The weights are public under a research-only license.

Why video depth flickers

Monocular depth is underconstrained because one camera view contains no direct measurement of scale. Models infer geometry from learned cues such as perspective, object size, texture, and occlusion. Small changes between video frames can therefore cause independent depth predictions to fluctuate, especially around hair, fur, and thin structures.

Frame-independent estimators often retain sharp contours while producing visible temporal jitter. Video-specific models reduce that instability by sharing information across frames, sometimes softening boundaries in the process. Depth Pro 2 uses one scalable architecture for both inputs. In its evaluation, Apple reports leading combinations of metric accuracy, boundary sharpness, temporal consistency, and runtime, which the paper describes as a new Pareto frontier.

Scatter plot comparing boundary sharpness and temporal consistency across depth models
Apple’s evaluation compares boundary accuracy with consistency between video frames.

Four checkpoints, one scale estimator

Apple provides four model sizes so developers can select a checkpoint for their memory, latency, and quality requirements. Every size can use a shared focal-length estimator when camera calibration or reliable EXIF data is unavailable. Focal length controls metric scale, so an incorrect value shifts every predicted distance by the same proportion.