Michigan Researchers' Point Prompting Turns Video AI Into a Zero-Shot Tracker
A red dot plus SDEdit turns any pretrained video generator into a zero-shot point tracker that survives occlusions and rivals self-supervised baselines.
ProTakeaways
- Point Prompting turns any pretrained video diffusion model into a zero-shot point tracker with no training required.
- Method: paint a red dot on frame one, regenerate the video with SDEdit, extract dot positions with color thresholding.
- Uses the unedited first frame as a negative prompt so the diffusion model does not erase the marker.
- Reaches 48.60 AJ on TAP-Vid DAVIS, beating all prior zero-shot methods and the self-supervised Opt-CWM baseline.
- Tracks reliably through occlusions thanks to the object-permanence prior baked into modern video generators.
- Paper and demos at point-prompting.github.io; accepted to ICLR 2026.
A team from the University of Michigan and Cornell has found something surprising hiding inside pretrained video diffusion models: they can track points through a video without any tracking-specific training. The trick is almost embarrassingly simple. Paint a red dot on the object you want to follow in the first frame, ask the model to regenerate the rest of the video, and watch the dot travel with the object, even through occlusions.
The method, called Point Prompting, was accepted to ICLR and comes from Ayush Shrivastava, Sanyam Mehta, Daniel Geng, and Andrew Owens. It reframes tracking as a counterfactual generation problem, and the resulting zero-shot tracker beats every previous zero-shot approach on the standard benchmark.
Why tracking has been stuck
Point tracking means following a specific pixel through a video as objects move, deform, and disappear behind other things. It sounds simple until you consider occlusion: when the point you are following vanishes behind a wall for two seconds, a good tracker needs a mental model of object permanence to know where it will reappear.
Modern trackers like CoTracker3 and TAPNext handle this well, but they need heavy supervision from labeled or synthetic data. Zero-shot approaches, which try to extract tracking behavior from models trained for something else, have historically been weak. The best of them, like DIFT and SD-DINO, treat image diffusion models as feature extractors and match patches between frame pairs, which leaves them unable to reason about a point when it is not visible.
Recent generative video models produce temporally consistent clips where objects persist across frames, through occlusion, and despite variations in camera pose and lighting, which is closely related to the visual tracking problem. The authors' question was whether tracking capabilities emerge automatically in video diffusion models, as a consequence of the close connection between the two problems.