Sonar on stock smartwatches leads to hand-tracking breakthrough (news.cornell.edu)

Researchers from Cornell University and KAIST have developed an innovative system called WatchHand, which harnesses AI-powered micro sonar technology integrated into standard smartwatches for continuous hand pose tracking. This groundbreaking approach allows users to control devices through discreet finger gestures—such as tapping their thumb and index finger—without requiring any additional hardware. The system utilizes the watch’s built-in speaker to emit inaudible sound waves that bounce off the hand, with its machine learning algorithm processing these acoustic echoes to accurately estimate hand movements in real time. The implications for the AI and machine learning community are significant. By making hand pose tracking accessible on existing smartwatches, WatchHand could fundamentally change interaction with technology, enabling gesture-based controls in various applications, including assistive tech for individuals with mobility challenges and immersive environments in AR and VR. The technology marks a departure from previous bulky prototypes, offering a more practical solution that prioritizes local data processing for enhanced privacy. However, while showing promising performance across varied conditions, WatchHand currently only operates with Android smartwatches, and its effectiveness diminishes during physical movement.