The release moves a technology Google has been developing as a research problem into consumer-facing software. SL2T was trained using more than 100,000 hours of material spanning over 50 sign languages, with ASL accounting for roughly one-quarter of the training data. According to DeepMind, training across multiple languages, dialects and levels of signing proficiency produced stronger results in its experiments than models trained around a single language.
Building an effective system requires more than recognizing individual hand gestures. Sign languages have their own vocabularies and grammatical structures, while meaning can be expressed through combinations of hand, arm, torso, head and facial movements. That makes the task both a computer vision problem and a machine translation problem rather than a direct conversion of gestures into corresponding English words.
Google's approach begins with MediaPipe Holistic running on the device. The system identifies landmarks on a signer and represents their movements as geometric coordinates. Those coordinates, rather than the original camera footage, are transmitted to the server for translation, according to the company. Google says the video itself can therefore be discarded immediately.
The translation model processes those landmark sequences and produces text without relying on an intermediate representation known as glosses. Such annotations have been widely used in previous sign-language translation research, but Google argues that they cannot fully represent features including spatial constructions and non-manual signals. Translating directly from body landmarks also avoids restricting the system to a predefined gloss vocabulary.
On the FLEURS-ASL sd-test benchmark for ASL-to-English translation, DeepMind reports that SL2T reached a zero-shot BLEURT score of 70. The company says that result exceeds previously reported scores on the benchmark. Its examples nevertheless show that the model can still make mistakes involving uncommon signs, fast fingerspelling, passive constructions, classifier depictions and tense when sufficient context is unavailable.
DeepMind says its work also extended beyond benchmark performance to problems likely to emerge during everyday use. The team focused on reducing delays during streaming translation and limiting hallucinations when the camera input contains no signing. It also worked on recognition for left-handed signers, who Google says represent 10% of signers, as well as people signing with one hand while using the other to hold a phone.
The project was developed with participation from members of the Deaf community. Google says Sam Sepah, a Deaf Googler, was involved in its conception, while Deaf partners contributed to data collection, user studies and assessments of the technology's potential impact.
DeepMind also created the AI Sign Language Advisory Committee, or AISLAC, with Deaf organizations and subject-matter experts participating in decisions around the technology's development. Google and the committee produced a joint impact report for the SL2T 1.0 release in Gboard and Live Transcribe that outlines both the system's capabilities and its current limitations.
The initial consumer release remains limited to ASL-to-English translation on Pixel 11. That leaves substantial room for expansion given Google's estimate that around 70 million Deaf and hard of hearing people use more than 200 sign languages worldwide. The company says it is working toward additional sign languages while also exploring sign-language generation and further AI capabilities.
For now, SL2T gives signing a direct role as an input method within Google's mobile software rather than requiring users to convert their communication into typed text first. The feature is available through Gboard and Live Transcribe on Pixel 11 at no additional cost, with Google saying support for more devices is coming soon.
This analysis is based on reporting from Google.
Image courtesy of Google.
This article was generated with AI assistance and reviewed for accuracy and quality.
