Qualcomm's Stream-VLM Watches You Exercise and Speaks up Without Being Asked
Qualcomm researchers built a live fitness coaching benchmark that exposes how badly current video language models handle proactive, real-time feedback.
ProTakeaways
- Qualcomm released QEVD, a 474-hour fitness coaching benchmark for real-time situated interaction with vision language models.
- Existing VLMs are turn-based and cannot proactively deliver timely feedback on unfolding video.
- Stream-VLM uses special
<next>and<feedback>tokens to decide when to speak versus keep watching. - Architecture pairs a motion-sensitive 3D CNN with a LLaMA-2-7B backbone through cross-attention layers.
- Stream-VLM beats fine-tuned turn-based baselines on both feedback quality and temporal F-score (0.56 vs 0.50).
- Code available at github.com/Qualcomm-AI-research/FitCoach with the dataset on Qualcomm's developer portal.
Most vision language models today wait to be spoken to. Feed them a clip, they answer a question, and the conversation stops until you prompt them again. A new benchmark and baseline from Qualcomm AI Research pushes on that limitation by asking a harder question: can a model watch you exercise in real time and know both what to say and when to say it?
The paper introduces QEVD, the Qualcomm Exercise Videos Dataset, along with Stream-VLM, a streaming video language model that decides on its own when to interrupt with corrective feedback. The benchmark requires models to recognize complex human actions, identify possible mistakes, and provide appropriate feedback in real time, which turns out to be something today's flagship open-source VLMs are genuinely bad at.
The turn-based ceiling
Situated interaction is the research term for conversations grounded in an unfolding physical scene. A fitness coach is the archetype: they watch, count reps, catch a sloppy squat mid-rep, and stay quiet when nothing needs saying. Current state-of-the-art models can be prompted to comment on events in a camera stream, but they cannot act asynchronously, speaking up when the situation demands rather than only when queried.
Fitness makes a clean testbed because routines are structured but the failure modes are rich. Users skimp on range of motion, break form, slow their pace, or stop moving entirely, and each mistake demands specific corrective language grounded in what the camera just saw.