DFKI's HandyLabel Lets Annotators Label Video Data With Hand Gestures
A gesture-driven annotation tool replaces click-heavy labeling workflows with hand signs, hitting 0.923 F1 and winning over 88.9% of testers.
PRO- HandyLabel replaces click-based video labeling with real-time hand gestures captured through a standard webcam.
- ResNet50 with MediaPipe skeleton preprocessing hit 0.923 F1 on the HaGRID dataset, beating raw-image baselines.
- User study with 46 participants: 88.9% preferred HandyLabel over Label Studio for annotation.
- Architecture streams 21 keypoints, not video, keeping latency under 100 ms and preserving privacy.
- Limited to five gestures, capping label vocabulary; validated reliably up to about three meters from camera.
- Open-source release available; full paper on arXiv.
Building a good machine learning model still starts with the least glamorous part of the pipeline: someone sitting in front of a screen, scrubbing through video, and clicking labels frame by frame. A team from DFKI and RPTU Kaiserslautern-Landau wants to replace that click-heavy ritual with a webcam and five hand gestures. Their system, HandyLabel, lets annotators throw a peace sign at the camera instead of pausing the timeline.
The core argument is that traditional tools like Label Studio and CVAT are built around post-processing. You record first, label later. That works for bounding boxes on static images, but it falls apart when the labels are things like emotion, engagement, or comprehension. By the time an annotator sits down to mark when a viewer looked confused, they are relying on memory of a moment that lasted a second or two, and memory is a lousy sensor for cognitive state.
Skeletons beat pixels
Before building the UI, the authors had to pick a gesture recognizer that could run on a normal laptop webcam without choking. They benchmarked ResNet18, ResNet50, ResNeXt50, MobileNetV3 (small and large), and ViT-B/16 against the publicly available HaGRID dataset, which contains over 500,000 annotated hand images across multiple gesture classes collected under diverse lighting and background conditions. The taxonomy was narrowed to five signs: Fist, Ok, Stop, Two-Up, and Peace.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
