---
title: "DFKI's HandyLabel Lets Annotators Label Video Data With Hand Gestures"
subtitle: "A gesture-driven annotation tool replaces click-heavy labeling workflows with hand signs, hitting 0.923 F1 and winning over 88.9% of testers."
url: "https://alphasignal.ai/news/dfki-s-handylabel-lets-annotators-label-video-data-with-hand-gestures"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-04-17T00:00:00.000Z"
updated: "2026-08-21T19:59:29.887Z"
type: "Paper"
topics:
  - "Data"
  - "Development"
subtopics:
  - "Data Curation"
upvotes: 2125
source_url: "https://arxiv.org/abs/2511.22337"
comments: 0
truncated: true
---

# DFKI's HandyLabel Lets Annotators Label Video Data With Hand Gestures

> A gesture-driven annotation tool replaces click-heavy labeling workflows with hand signs, hitting 0.923 F1 and winning over 88.9% of testers.

Building a good machine learning model still starts with the least glamorous part of the pipeline: someone sitting in front of a screen, scrubbing through video, and clicking labels frame by frame. A team from DFKI and RPTU Kaiserslautern-Landau wants to replace that click-heavy ritual with a webcam and five hand gestures. Their system, [HandyLabel](https://arxiv.org/abs/2511.22337), lets annotators throw a peace sign at the camera instead of pausing the timeline.

The core argument is that traditional tools like Label Studio and CVAT are built around post-processing. You record first, label later. That works for bounding boxes on static images, but it falls apart when the labels are things like emotion, engagement, or comprehension. By the time an annotator sits down to mark when a viewer looked confused, they are relying on memory of a moment that lasted a second or two, and memory is a lousy sensor for cognitive state.

![Comparison of post-processing annotation versus HandyLabel real-time gesture annotation](https://arxiv.org/html/2511.22337v2/teaser.png) 

## Skeletons beat pixels

Before building the UI, the authors had to pick a gesture recognizer that could run on a normal laptop webcam without choking. They benchmarked ResNet18, ResNet50, ResNeXt50, MobileNetV3 (small and large), and ViT-B/16 against the publicly available HaGRID dataset, which contains over 500,000 annotated hand images across multiple gesture classes collected under diverse lighting and background conditions. The taxonomy was narrowed to five signs: Fist, Ok, Stop, Two-Up, and Peace.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [HandyLabel: Towards Post-Processing to Real-Time Annotation Using Skeleton Based Hand Gesture Recognition](https://arxiv.org/html/2511.22337)
- [HaGRID - HAnd Gesture Recognition Image Dataset](https://arxiv.org/abs/2206.08219)
- [HaGRID GitHub repository](https://github.com/hukenovs/hagrid)
- [MediaPipe](https://mediapipe.dev/)
- [Label Studio](https://github.com/heartexlabs/label-studio)