It works by using AI to monitor a live video feed. The AI system has been trained to recognize hand gestures and their associated letters. When a letter has been confirmed, the Pi uses text-to-speech to say the letter aloud. This makes it possible for anyone nearby to understand what’s being conveyed, whether or not they are familiar with any sign language at all.
The platform holding together the design is VIAM, an open-source tool that’s oriented toward smart machine projects like this one. Because this project is so reliant on AI and image processing, it’s understandable that Nekhil would first turn to the latest Raspberry Pi 5. However, it quickly became evident that he could get away with using a Pi Zero 2 W, which would be plenty capable performance-wise while providing a smaller form factor.
The Raspberry Pi Zero 2 W is connected to a Camera Module V3, which is positioned in the front center of the glasses frame. This makes it possible to capture images and video of whatever is directly in front of the wearer. As long as you’re facing the person who’s signing, the Pi will be able to read the gestures. According to Nekhil, the frames were designed using Fusion 360 and 3D printed just for this project.
In the project details, Nekhil explains that the machine learning model vision service that VIAM uses is able to work with Tensor Flow Lite models. However, this project is using a YOLOv8 model. Although American Sign Language (ASL) is a vast language with many signs for words and phrases, this system is designed to recognize only letters for spelling.
If you want to get a closer look at this Raspberry Pi project, you can check it out over at Hackster and see it in action at the YouTube video embedded above.
