VPTracker Lets an AI Draw on Video Frames to Track Anything Globally
A new tracking framework uses Qwen3-VL to search the entire image, with a drawn rectangle nudging the model toward where the target was last seen.
PRO- VPTracker is the first vision-language tracker built directly on a Multimodal Large Language Model (paper).
- It searches the entire frame every step instead of cropping a local window around the last prediction.
- A rectangle drawn on the image tells the MLLM where to look first before falling back to global search.
- Built on Qwen3-VL-4B, trained on TNL2K plus TNLLT with roughly one million samples on 8 H200 GPUs.
- Sets new SOTA: 64.9 AUC on TNL2K and 62.5 SR on TNLLT, beating prior VL trackers.
- Code and models are open source at github.com/jcwang0602/VPTracker.
Object tracking has spent years chasing a small window around wherever the target was last spotted. It works, until the subject sprints, hides, or the camera swings. A new paper called VPTracker throws that assumption out and asks a Multimodal Large Language Model to look at the whole frame every time, using a simple trick to keep it from getting distracted.
The result is the first global vision-language tracking framework built on an MLLM backbone, and it posts state-of-the-art numbers on two long-form tracking benchmarks. The code and models are already public.
Why local search keeps breaking
Vision-language tracking (VLT) follows an object through a video given two things in the first frame: a bounding box crop of the target (the visual template) and a natural language description like "the girl in red uniform with fifteenth number." Existing methods are typically limited to local search, making them prone to failures under viewpoint changes, occlusions, and rapid target movements.
The reason is mechanical. Most trackers crop a small search window centered on the previous predicted box, then run matching inside that window. That keeps compute low and background clutter out, but if the subject leaves the window, the tracker has no way to find it again. MLLMs exhibit strong global scene understanding and cross-modal reasoning, which suggests they could reshape long-term tracking by performing global search over the entire image rather than relying on local heuristics.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
