Gemini-2.5-Pro Tops Sign Language Benchmark but Still Trails Experts by 65 Points

A group of researchers just ran the experiment a lot of people have been quietly wondering about: can a modern vision language model watch a short clip of someone signing and tell you what the sign means, with no fine-tuning at all? The paper benchmarks a dozen open-source and proprietary VLMs on isolated sign language recognition and finds that most are near random, while the frontier proprietary models show real but limited understanding. The code and evaluation harness are on GitHub.

The task nobody trained for

Isolated Sign Language Recognition (ISLR) processes short video clips containing a single sign and predicts the corresponding gloss label, which is the English word conventionally used to write down that sign. Traditional systems rely on task-specific supervised learning, requiring large annotated datasets and specialized architectures to model hand motion, body pose, and temporal dynamics. They generally do not generalize well across signers or environments.

VLMs are the natural candidate to change that. They already handle zero-shot visual reasoning, and if they can recognize signs out of the box, an entire pipeline of pose extractors, graph convolutions and gloss classifiers could collapse into a prompt. The authors set out to measure exactly how far off that future is.

How the benchmark is set up

The evaluation runs on WLASL300, a widely adopted ISLR benchmark that offers a practical compromise between computational cost and label-set diversity: 300 glosses, large enough to meaningfully probe open-vocabulary behavior while remaining tractable for multi-model, multi-prompt evaluation. The test split has 668 clips. Every model gets the same system prompt telling it to act as an ASL gloss annotator and output a single English word.