RegVGGT Lets One Consumer GPU Process Thousands of 3D Video Frames
A training-free trick keeps only the top 1% of tokens per frame, letting VGGT-style models reconstruct thousands of video frames on a single GPU.
- RegVGGT admits only the top 1% of tokens per frame into memory, keeping streaming 3D reconstruction feasible on one GPU.
- Training-free method built on VGGT, no fine-tuning required to deploy.
- Key insight: a token's early-layer saliency predicts its long-term importance across the stream.
- Uses a FlashAttention-compatible scoring scheme, preserving fused-attention speed and memory benefits.
- State-of-the-art on TUM Dynamics, 7-Scenes, Sintel and KITTI long-horizon benchmarks versus StreamVGGT, Evict3R, InfiniteVGGT.
- Scales to thousands of frames where baselines like StreamVGGT cap out near 200.
Feed-forward 3D reconstruction models such as VGGT infer camera parameters, depth maps, point maps, and 3D point tracks directly from one or more images. They can replace parts of traditional structure-from-motion pipelines, which match features across images and refine camera and scene estimates through costly bundle adjustment.
VGGT was designed around offline image batches. Long video streams expose a memory problem because each frame adds image tokens that future frames may need. The RegVGGT preprint introduces a training-free token filter that sharply reduces this growth, allowing the model to process thousands of frames on a single consumer GPU, according to the authors.
Video overwhelms the attention cache
Each frame is divided into patch tokens. During transformer inference, the model stores key and value representations, commonly called a KV cache, so later tokens can attend to earlier observations. Retaining every representation preserves scene history, but memory consumption rises with every frame.
| Memory strategy | Growth | Main trade-off |
|---|---|---|
| Full explicit history | All tokens accumulate | Preserves context but quickly exhausts GPU memory |
| Implicit summary | Bounded hidden state | Can forget geometry observed earlier |
| Fixed explicit cache | Capped token budget | Depends on eviction rules and assumptions about scene scale |
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
