GAASH Lab's QTrack Tracks Any Video Subject Using Plain English Descriptions
QTrack turns multi-object tracking into a language-conditioned reasoning task, hitting state-of-the-art on a new RMOT26 benchmark with just 3B parameters.
- QTrack reframes multi-object tracking as language-conditioned spatiotemporal reasoning via a new paper and open-source code.
- A 3B model beats GPT-5.2, Qwen3-VL 8B and Gemma 3 27B on the RMOT26 benchmark.
- Introduces TPA-PO, a GRPO-style policy optimization with motion-aware structured rewards.
- RMOT26 uses sequence-level splits to prevent identity leakage across train and test.
- Weights and dataset available on Hugging Face under Apache-2.0.
- Fails without initial bounding box; some sequences collapse to repeating the first box.
QTrack lets an application request “the person in the red jacket carrying a backpack” and receive that subject’s bounding-box trajectory across a video. The 3B-parameter vision-language model combines semantic grounding with temporal association, placing target selection inside the tracking task.
GAASH Lab describes the system in the QTrack paper, titled QTrack: Query-Driven Reasoning for Multi-modal MOT. The release contains three components:
- QTrack: an end-to-end model for query-conditioned tracking.
- RMOT26: a benchmark with grounded queries and sequence-level data splits.
- TPA-PO: a reinforcement-learning method designed to reward motion-aware predictions.
A prompt narrows the track
Conventional multi-object trackers such as SORT, ByteTrack, and MOTR detect objects in configured classes and assign persistent IDs across frames. Applications that need one semantically defined subject must add another selection layer after tracking.
Descriptions such as “the child in the blue shirt” require more than position and motion. The tracker must connect words to visual attributes, distinguish similar-looking subjects, and preserve the selected identity through occlusion and camera movement.
QTrack consumes a reference frame, a video sequence, and a natural-language query. It returns bounding-box trajectories for the requested target or targets, along with a structured reasoning trace. Its reported performance relies heavily on an initial bounding box in the reference frame, a constraint that affects how developers can integrate it.
Grounding, association, and prediction share one model
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
