AI tool helps visually impaired users 'feel' where objects are (www.psu.edu)

Penn State researchers unveiled NaviSense, a smartphone app that helps people with visual impairments “feel” where objects are by combining voice prompts, real-time vision, and tactile/auditory guidance. Presented at ACM SIGACCESS ASSETS ’25 (Best Audience Choice Poster) and described in the conference proceedings, NaviSense listens to a user’s spoken request, uses the phone camera to scan the scene, and guides the user to the requested object with audio cues and vibrations. Test users reported a better experience than existing assistive options, while early design was driven by interviews with visually impaired people to prioritize privacy, convenience, and real-world needs. Technically, NaviSense integrates vision-language models (VLMs) and large language models (LLMs) hosted on an external server so it can recognize and reason about objects on-the-fly rather than relying on preloaded object models. The system filters irrelevant items, asks follow-up questions when queries are ambiguous, and supports flexible, conversational searches. This approach improves generalization to novel objects and use cases, increasing autonomy and scalability compared with human-guided services that raise privacy and latency concerns. Key implications include enhanced independence for users and a path toward more adaptive assistive apps, though reliance on external servers highlights trade-offs around connectivity, latency, and on-device privacy.