---
title: "Meta's PWR System Watches You Cook and Catches Mistakes Before You Ask"
subtitle: "Researchers release PWR, a benchmark and dual-model architecture that teaches AI assistants to watch, plan, and recover when users go off-script during real-world tasks"
url: "https://alphasignal.ai/news/meta-s-pwr-system-watches-you-cook-and-catches-mistakes-before-you-ask"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-06-03T00:00:00.000Z"
updated: "2026-06-11T05:39:20.615Z"
type: "Paper"
topics:
  - "Video"
  - "Data"
subtopics:
  - "Video Understanding"
  - "Datasets"
upvotes: 4427
source_url: "https://arxiv.org/abs/2606.04970"
comments: 0
truncated: true
---

# Meta's PWR System Watches You Cook and Catches Mistakes Before You Ask

> Researchers release PWR, a benchmark and dual-model architecture that teaches AI assistants to watch, plan, and recover when users go off-script during real-world tasks

Most AI assistants wait to be asked. The real challenge is building one that watches you work, knows when you've made a mistake, and tells you how to fix it , all without you saying a word. A team of researchers from Meta Reality Labs and Meta Superintelligence Labs has published [Plan, Watch, Recover (PWR)](https://arxiv.org/abs/2606.04970), a paper introducing a benchmark, a dataset, and an architecture designed to make that vision real.

The work is a multi-institutional research contribution led by Kaustav Kundu and Ritvik Shrivastava, with 16 authors in total. It is not a product release , it is a research paper that ships a dataset, a benchmark suite, and a training recipe that other teams can build on.

## The problem nobody was measuring

Building a proactive procedural assistant requires solving three things at once: tracking where a user is in a multi-step task, deciding **when** to speak up, and generating **what** to say , including how to recover when the user goes off-script. Prior work addressed these in isolation. The deeper problem was that no benchmark existed to evaluate all three together, especially in the realistic case where users deviate from the expected sequence.

The authors identify three concrete barriers that block deployment:

* **No benchmark covers user deviations.** Existing egocentric corpora assume canonical execution. Real users skip steps, reorder them, and make mistakes , what the paper calls Out-of-Plan (OOP) events. No public dataset paired these deviations with recovery guidance or per-decision-point intervention labels under wearable capture.
* **Real-time interaction and long-horizon planning have conflicting compute budgets.** Per-frame streaming decisions need to be fast; reasoning about a full procedural plan is slow. Monolithic models are forced to trade one against the other.
* **Post-training gains are model-specific.** Existing recipes are tied to single architectures and don't transfer.

## Four contributions in one paper

PWR addresses all three barriers with four simultaneous contributions. The first is **EgoProactive**, a new wearable-first dataset recorded on Ray-Ban Meta smart glasses. It contains 9,935 evaluation instances from 700 egocentric videos across four activity domains (cooking, crafts, DIY, and tutorials). Crucially, 1,833 of those instances include scripted Out-of-Plan deviations , step omissions, wrong orderings, and substitution errors , each paired with ground-truth recovery guidance. It is the first dataset to provide wearable-form-factor captures with both per-decision-point intervention labels and paired deviation–recovery sequences.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Plan, Watch, Recover: A Benchmark and Architectures for Proactive Procedural Assistance](https://arxiv.org/abs/2606.04970)
- [PWR Paper Full HTML](https://arxiv.org/html/2606.04970)
- [EgoProactive Dataset on Hugging Face](https://huggingface.co/datasets/facebook/wearable-ai)