Google DeepMind's SL2T Doubles Sign Language AI Accuracy, Giving Deaf Users a Voice

For decades, voice-to-text has been a standard feature on every smartphone. Deaf and hard of hearing users have had no equivalent. That changes today: Google DeepMind has announced SL2T (Sign Language to Text), a massively multilingual sign language translation model that powers new sign-to-text dictation features in Gboard and Live Transcribe, launching first on the Pixel 11 family with American Sign Language (ASL) to English support.

What it does and how to use it

Google has embedded the model inside Gboard and Live Transcribe, where it allows Deaf and hard of hearing users to sign to their phone anywhere they would have normally typed. That means signing to search the web, draft messages, ask Gemini a question, or respond in a Live Transcribe conversation, all without touching the keyboard. In practice, this gives users a faster, more natural way to search the web, write messages, and converse with Google's Gemini chatbot.

Sign-to-Text lets you sign into the front camera while the viewer sees the transcription on the outer display -- a particularly useful setup for the Pixel 11 Pro Fold. The feature is available at no additional cost, and Google plans to expand it to more Android devices beyond the Pixel 11 line.

The technical challenge nobody solved until now

Sign language AI has been a notoriously hard problem, and not just for the obvious reasons. AI's ability to process spoken languages has advanced rapidly over recent decades, enabling automatic translation, dictation, and conversational interfaces that feel effortless to hearing users -- yet this technological revolution has not reached the world's more than 200 sign languages and the estimated 70 million Deaf and hard of hearing people who use them.

Two core problems have blocked progress. First, sign languages are not simply encoded versions of spoken languages. Transcribing speech is a matter of performing a sequential mapping from sound to text in the same language, whereas sign languages are independent, natural languages with their own distinct grammars and lexicons -- requiring true machine translation rather than a sequential process of sign-to-word transformations. Second, the model must learn to see and understand physical movement: sign languages convey meaning through simultaneous movements of the hands, arms, torso, head, and face, and accurately tracking these at high frame rates is a difficult and computationally demanding computer vision task.

This is why early approaches like sign language gloves were dead ends. They assumed sign language was just English mapped onto hand positions, which it is not.

How SL2T is built

DeepMind trained SL2T on more than 100,000 hours of multilingual sign language data, with approximately a quarter of the dataset representing American Sign Language. Training jointly across languages is a deliberate architectural choice: training jointly on diverse languages, dialects, and proficiency levels causes the model to learn shared underlying structures, outperforming single-language models in experiments.

The privacy architecture is worth understanding. SL2T doesn't interpret raw video. Instead, a separate on-device model converts footage into a wireframe of geometric coordinates that are sent to Google's servers -- a design choice intended to protect user privacy. That on-device model is MediaPipe Holistic, which tracks landmarks across the face, hands, and body. Only these coordinate sequences leave the device; the original video is discarded immediately.

The other key architectural decision is going gloss-free. Glosses are word-like annotations that researchers have historically used as an intermediate step between signing and text -- essentially a simplified written representation of individual signs. SL2T differs from past sign-language-to-text systems in that it translates the wireframes it receives directly into text, instead of producing intermediate glosses. As DeepMind explains, glosses fail to capture rich, non-linear aspects of sign languages such as non-manual markers and spatial constructions. Skipping this step removes artificial vocabulary limits and lets translation quality scale directly with data volume.

Benchmark performance: a significant leap

SL2T is the most capable sign language translation model to date according to key benchmarks like FLEURS-ASL (sd-test), which assesses ASL to English translation quality. SL2T achieves a remarkable zero-shot score of 70 BLEURT, which is significantly higher than any previously reported score. For context, BLEURT (Bilingual Evaluation Understudy with Representations from Transformers) is a learned metric that measures semantic similarity between a translation and a reference -- it correlates better with human judgment than older word-overlap metrics like BLEU. Prior academic work on FLEURS-ASL was scoring in the high 30s to low 40s BLEURT range, making SL2T's 70 a substantial jump.

The team also engineered for real-world use, not just benchmark performance. That included:

• Minimizing streaming latency so translation feels live
• Preventing hallucinations when the camera sees non-signing inputs
• Ensuring fairness for left-handed signers (roughly 10% of the population)
• Improving accuracy for one-handed signing, which is common when holding a phone

Where it falls short

The blog post is transparent about current limitations. The model only supports ASL to English transcription at launch. The translation examples provided also reveal specific failure modes: rapid fingerspelling (spelling out individual letters) can trip the model up -- the blog shows "prey" being transcribed as "grey" -- and passive constructions, classifier depictions, and tense without context are areas where errors still appear. These are hard linguistic problems, not engineering oversights.

Built with the Deaf community

The project's governance model is notable. DeepMind established the AI Sign Language Advisory Committee (AISLAC), bringing together global Deaf organizations and subject-matter experts to guide responsible real-world deployment. The team also co-authored a joint impact report with AISLAC, transparently detailing capabilities and current limitations -- a participatory approach they plan to continue for all major sign language releases. The project was originally conceptualized by Sam Sepah, a Deaf Googler.

This matters because the broader research community has flagged real concerns here. Sign language translation systems have been met with strong skepticism from Deaf communities due to risks of misrepresenting and oversimplifying the richness of signed communication. DeepMind's community-first approach is a direct response to that history.

What this changes for the field

The assumptions that need updating are significant. For years, the field treated sign language AI as a niche research problem with limited data and unclear deployment paths. SL2T demonstrates that:

• Massive multilingual training (100,000+ hours, 50+ sign languages) is achievable and effective
• Gloss-free, landmark-based translation can reach production quality at scale
• Privacy-preserving architectures (pose landmarks only, no raw video to servers) are viable for real-time use
• A single model trained across many sign languages can outperform specialized single-language models

Globally, there are more than 70 million Deaf and hard of hearing people who communicate in some 200 sign languages. The good news is that SL2T was simultaneously trained on multiple languages so that it could learn the shared underlying structures between them -- meaning the path to supporting more languages is not starting from scratch each time.

The team is already working on sign language generation (text-to-sign), expansion to additional sign languages, and integration with frontier AI capabilities. For now, ASL on Pixel 11 is the starting line, not the finish.