Researchers have developed a system called WatchHand that turns ordinary smartwatches into real-time hand-tracking devices using AI-powered sonar. Instead of relying on cameras or extra sensors, the smartwatch emits inaudible sound waves through its speaker; these waves bounce off the user’s hand and are captured by the microphone. A machine-learning model processes the returning echo profile to reconstruct the hand’s position and finger movements in 3D, in real time, all directly on the device.
This approach is significant because it works on off-the-shelf smartwatches without additional hardware, making it scalable and practical for everyday use. Tests with participants showed it can reliably track gestures like finger movements and wrist rotations, enabling applications such as gesture-based control of computers, AR/VR interaction, and assistive technologies. The system also preserves privacy by processing data locally, though it still has limitations, such as reduced accuracy while the user is moving and current compatibility mainly with Android devices.
More information:
https://interestingengineering.com/innovation/ai-smartwatch-hand-tracking-sonar-watchhand
