Meta's PWR System Watches You Cook and Catches Mistakes Before You Ask
Researchers release PWR, a benchmark and dual-model architecture that teaches AI assistants to watch, plan, and recover when users go off-script during real-world tasks
- What: Researchers publish PWR (Plan, Watch, Recover), a benchmark + architecture for AI assistants that proactively guide users through multi-step tasks in real time.
- Dataset: EgoProactive — 9,935 eval instances from 700 smart-glasses videos, with 1,833 scripted Out-of-Plan deviations paired with recovery guidance; available on Hugging Face.
- Benchmark: Pro²Bench unifies 5 existing datasets (Ego4D, EPIC-KITCHENS, EgoExo4D, HoloAssist, HowTo100M) into 42,275 eval instances under a shared proactive-guidance schema.
- Architecture: A decoupled Duplex Interaction Model (watches video at 2fps, decides silent/interrupt) + a Planner (runs only on interrupts, updates structured plan or generates recovery steps).
- Results: Trained PWR reaches 0.83 G-Mean F1 vs. ≤0.51 for all zero-shot frontier models; oracle upper bound hits 99.6% OOP detection recall and 4.82/5 recovery quality.
- Key finding: Giving a plan to a zero-shot model makes timing worse 80% of the time; the architecture only pays off after task-specific training on the new data.
Most AI assistants wait to be asked. The real challenge is building one that watches you work, knows when you've made a mistake, and tells you how to fix it , all without you saying a word. A team of researchers from Meta Reality Labs and Meta Superintelligence Labs has published Plan, Watch, Recover (PWR), a paper introducing a benchmark, a dataset, and an architecture designed to make that vision real.
The work is a multi-institutional research contribution led by Kaustav Kundu and Ritvik Shrivastava, with 16 authors in total. It is not a product release , it is a research paper that ships a dataset, a benchmark suite, and a training recipe that other teams can build on.
The problem nobody was measuring
Building a proactive procedural assistant requires solving three things at once: tracking where a user is in a multi-step task, deciding when to speak up, and generating what to say , including how to recover when the user goes off-script. Prior work addressed these in isolation. The deeper problem was that no benchmark existed to evaluate all three together, especially in the realistic case where users deviate from the expected sequence.
The authors identify three concrete barriers that block deployment:
- No benchmark covers user deviations. Existing egocentric corpora assume canonical execution. Real users skip steps, reorder them, and make mistakes , what the paper calls Out-of-Plan (OOP) events. No public dataset paired these deviations with recovery guidance or per-decision-point intervention labels under wearable capture.
- Real-time interaction and long-horizon planning have conflicting compute budgets. Per-frame streaming decisions need to be fast; reasoning about a full procedural plan is slow. Monolithic models are forced to trade one against the other.
- Post-training gains are model-specific. Existing recipes are tied to single architectures and don't transfer.
Four contributions in one paper
PWR addresses all three barriers with four simultaneous contributions. The first is EgoProactive, a new wearable-first dataset recorded on Ray-Ban Meta smart glasses. It contains 9,935 evaluation instances from 700 egocentric videos across four activity domains (cooking, crafts, DIY, and tutorials). Crucially, 1,833 of those instances include scripted Out-of-Plan deviations , step omissions, wrong orderings, and substitution errors , each paired with ground-truth recovery guidance. It is the first dataset to provide wearable-form-factor captures with both per-decision-point intervention labels and paired deviation–recovery sequences.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
