Skip to main content
Back

KIT Maps 90 Deep Learning Methods That Predict Human Actions Before They Happen

A comprehensive taxonomy of over 90 deep learning methods for predicting human actions before they happen, with benchmark results across major datasets.

Pro

Paper, Full Access
  • Survey categorizes 90+ deep learning action anticipation methods into five technical groups.
  • Field has converged on predicting future feature representations rather than raw pixels for tractability.
  • PlausiVL leads EpicKitchens-100 with 27.6 action recall using ViT-G/14 plus an LLM.
  • Recognition quality is the main bottleneck for long-term anticipation on Breakfast dataset.
  • LLM-based methods like PALM and BiAnt dominate Ego4D long-term anticipation benchmarks.
  • Open gaps: real-time latency, multi-person scenes, personalization, and uncertainty evaluation protocols.

Predicting what a person is about to do, before they do it, is one of the more slippery problems in computer vision. A wearable assistant that hands you a whisk before you reach for it, a self-driving car that brakes because a pedestrian is about to step off the curb, a warehouse robot that stays clear of a worker's swing path all depend on this kind of anticipation. A new survey pulls together over 90 deep learning methods that tackle the task, organizes them into a coherent taxonomy, and reports benchmark results across the major datasets in the field.

The survey from researchers at Karlsruhe Institute of Technology, Fraunhofer IOSB, and the University of Bonn focuses on daily-living scenarios, categorizes methods by their primary technical contribution, and offers a rare side-by-side comparison of architectures, modalities, and training regimes across a fragmented literature.

Why anticipation is harder than recognition

Human action recognition and early action recognition both let a model observe some or all of the action itself. Anticipation predicts a future action without observing any part of it. Recognition models can lean on the frame they are looking at. Anticipation has to reason about what has not happened yet, which is often non-deterministic.

Timeline diagram contrasting recognition, early recognition, and anticipation tasks

Predictability itself varies wildly by domain. Industrial assembly and medical procedures follow tight protocols and are highly predictable. Cooking sits in the middle: recipes constrain the order, but individual choices scramble it. Fully spontaneous behavior is hardest, since it arises without explicit stimuli or established patterns.