Skip to main content
Back

SOAP-Net Beats SOTA by 6 Points on Few-Shot Video Recognition

A plug-and-play module for few-shot action recognition tackles high frame-rate video by capturing motion across variable-length frame tuples.

Pro

  • SOAP is a plug-and-play module for few-shot action recognition that improves motion capture in high frame-rate video.
  • Three parallel modules (3DEM, CWEM, HMEM) add spatio-temporal and motion priors before feature extraction.
  • HMEM captures motion across variable-length frame tuples via sliding windows, not just adjacent frames.
  • SOAP-Net sets new SOTA on SthSthV2, Kinetics, UCF101, and HMDB51 with both ResNet-50 and ViT-B backbones.
  • Plugged into TRX, HyRSM, and MoLo it adds 6-8 points on Kinetics without changing their backbones.
  • Code and training scripts available at github.com/wenbohuang1002/SOAP under MIT license.

Action recognition models are hungry for labeled video data, and that appetite grows as cameras keep improving. A new paper accepted at ACM Multimedia argues that the very thing that makes modern video look good, high frame rates, also makes it harder to learn from small datasets. The authors propose SOAP, a plug-and-play architecture that rethinks how spatial, temporal, and motion signals are combined for few-shot action recognition.

The high frame-rate trap

Progress in video recorders keeps pushing frame rates up, and packing more similar frames into each second helps express fine-grained actions. Spatio-temporal relations and motion density decrease as fluency improves. When frames are packed close together in time, the change from one to the next becomes tiny, which weakens the motion cues a model relies on. That gets patched over by throwing more labeled videos at the problem, but samples for target actions such as "falling down" are usually insufficient and hard to collect in real-world scenarios.

Few-shot action recognition (FSAR) tries to classify a query clip using just a handful of labeled support clips per class, typically in a 5-way 1-shot or 5-shot setup. The authors point out two recurring weaknesses in prior FSAR work. Existing methods prioritize temporal alignment after spatial feature extraction, cutting apart spatial and temporal features within samples and overlooking motion information capture. When motion is considered, it is usually computed between adjacent frames only, missing the bigger picture.