Skip to main content
Back

Google DeepMind's TAPNext++ Tracks Any Pixel for 4,000 Frames at 562 FPS

A Google DeepMind team extends the TAPNext recurrent tracker from 150 frames to 4,000+ while adding robust re-detection after occlusion.

Pro

  • TAPNext++ extends recurrent point tracking from ~150 frames to 4,000+ without changing the architecture (paper).
  • Distributed parallel scan across 8 GPUs enables end-to-end training on 1,024-frame video sequences.
  • Roll augmentation wraps frames around boundaries, forcing appearance-based re-detection instead of spatial-proximity shortcuts.
  • New AJ_RD metric measures tracking quality specifically after a point reappears from occlusion or off-screen.
  • Achieves 72.2 mean δ_avg across five benchmarks and 562 FPS on H100, best online result on record.
  • Checkpoints, Colab demos, and code available in the google-deepmind/tapnet repository.

Tracking a single pixel through a video sounds like a solved problem until the pixel disappears behind an object, drifts off the edge of the frame, or the clip runs longer than a minute. That is where most online point trackers fall apart. A team from Google, DeepMind, DLR, Mila, and TUM has released TAPNext++, a fine-tuned version of the recurrent TAPNext architecture that keeps points locked in for thousands of frames and reliably re-detects them after they vanish, without changing the model architecture or inflating its compute footprint.

The paper is a CVPR 2026 Findings entry, and unusually for a tracking paper its contribution lives almost entirely on the training and evaluation side: a distributed parallel scan that unlocks 1,024-frame training, a synthetic dataset built for long-horizon dynamics, a geometric augmentation that forces appearance-based re-identification, and a new metric that measures what everyone else was quietly ignoring.

Why recurrent trackers were hitting a wall

Tracking Any Point (TAP) is the task of following an arbitrary query pixel through a video, reporting both its 2D position and whether it is visible in each frame. It underpins AR anchoring, robotic manipulation, and 4D reconstruction. The original TAPNext reframed this as next-token prediction and used a recurrent video transformer with alternating ViT and State-Space-Model blocks, giving it constant-memory inference and letting it track 1024 points at 348 FPS on an H100.

The catch: TAPNext was trained on 48-frame clips. Push it past roughly 150 frames and its recurrent state degrades. When a query point leaves the frame and re-enters somewhere else, the model tends to lose it entirely. The authors demonstrate this by feeding the same static image over and over into the pretrained BootsTAPNext checkpoint. After about 100 frames, the norm of the LRU cache state starts drifting erratically, a hint that whatever the model "remembers" about the initial query becomes corrupted.