Edge Hill's TAG-Head Beats Multimodal AI on Video Recognition Using Only RGB
A lightweight Transformer plus graph module lets RGB-only video models beat multimodal systems that rely on pose and text annotations.
ProTakeaways
- TAG-Head is a plug-and-play graph head that upgrades 3D video backbones for fine-grained action recognition using RGB only.
- Combines a small Transformer for global context with a graph using intra-frame fully-connected and time-aligned temporal edges.
- Uses APPNP (personalised PageRank) propagation, which is parameter-free and avoids oversmoothing from stacked GCN layers.
- Hits 95.6% on Gym99, 92.2% on Gym288, and 86.1% on HAA500, beating multimodal systems using pose and text.
- Only 69.9M parameters total, 3.8x smaller than PGVT and 15.7x smaller than PEVL at similar or better accuracy.
- Paper accepted at ICPR 2026; code promised on GitHub.
Telling apart a gymnast doing a switch leap with a half turn from one doing a full turn has almost nothing to do with what is in the frame. The bodies, floor, and lighting look identical. What separates the classes is rotation angle over a few frames. This is the core problem in fine-grained human action recognition (FHAR), and it explains why recent leaderboards have been dominated by heavy multimodal systems that pull in pose skeletons, text prompts, or optical flow to give the model extra signal.
A team from Edge Hill University is pushing back. Their new module, TAG-Head, is a slim head that bolts onto standard 3D video backbones like SlowFast, I3D, or R(2+1)D and beats several multimodal state-of-the-art systems using only RGB frames. The paper was accepted at the 28th International Conference on Pattern Recognition.

The multimodal tax
Most recent gains on fine-grained benchmarks come from fusing video with something else. Pose-guided video transformers like PGVT inject skeleton streams, while pose-enhanced vision-language models like PEVL layer in text descriptions alongside pose. These work, but they push cost onto deployment: you now need a robust pose estimator at test time (which breaks under occlusion or camera motion), plus cross-modal fusion machinery, plus annotation pipelines for text.
For broadcast sports, CCTV, or consumer devices, that overhead is a dealbreaker. The authors argue that RGB pipelines have not been getting a fair shake, and that shortcomings such as over-reliance on scene context, weak long-range temporal modelling, and limited capacity to resolve subtle spatiotemporal differences between visually similar actions expose a gap between general video understanding and the specific demands of FHAR.