Skip to main content
Back

Qualcomm's LiveMamba Watches You Cook and Catches Mistakes in Real Time

Qualcomm's new benchmark and streaming Mamba-based model tackle real-time cooking coaching, exposing where current multi-modal LLMs fail at watching users work.

Pro

  • Qualcomm introduces Interactive Cooking, a benchmark for live, timestamped step-by-step task guidance with mistake alerts.
  • Built on CaptainCook4D with new dense annotations for instructions, completions, and precisely-timed mistake feedback.
  • LiveMamba pairs InternViT plus Q-Former with a Mamba-130M backbone for on-device streaming inference.
  • An external re-planner handles graph-structured recipes when users make divergent mistakes and branches change.
  • Counterfactual mistake and instruction-completion augmentations drive most of the gains, exposing the real bottleneck: data.
  • Dataset on Hugging Face, evaluation code on GitHub.

Ask a multi-modal model to describe a cooking video and it does fine. Ask it to sit next to you at the stove, tell you the next step at the right moment, notice when you salt the wrong bowl, and warn you before you ruin dinner, and it falls apart. A new benchmark and model from Qualcomm AI Research puts numbers on that gap and offers a first attempt at closing it.

The paper, Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance?, introduces two contributions: the Qualcomm Interactive Cooking dataset and benchmark, and LiveMamba, a streaming vision-language model built around a tiny Mamba backbone. Together they target a capability that AR glasses and embodied assistants keep promising but rarely deliver: asynchronous, real-time coaching that watches a live video feed and speaks up at the correct instant.

Why current MLLMs choke on live coaching

Standard multi-modal LLMs are turn-based. You send a message, they respond. Live guidance works differently. Effective coaching requires delivering instructions, detecting their successful execution, and alerting users to mistakes, all in real-time. That demands models that react asynchronously to a video stream, plus training footage of users performing tasks including mistakes and their corrections.

Two problems compound. First, the model has to decide when to speak, not just what to say. Interrupting too early is annoying; too late is useless. Second, the training data for this simply does not exist at scale. Existing video-instruction datasets show people doing things correctly, not fumbling and recovering.

Building a benchmark with mistakes in it

The team started from CaptainCook4D, which contains 384 videos of a person preparing a dish from an egocentric view, each associated with a graph-structured recipe and annotated with temporal action segments. If an action contains a mistake, there is a description of the mistake. However, there is no timestamp annotation for the mistake.