Driving directions from LLM memory (fig.blue)

Recent testing of various local and proprietary large language models (LLMs) revealed significant disparities in their ability to provide accurate turn-by-turn directions from memory. The evaluations involved scenarios set in familiar environments, ranging from Austin to Bermuda, and included complex routing tasks, such as multi-stop trips. The Opus 5.5 model performed the best, scoring notable points in several scenarios, while Haiku 5.5 and Qwen struggled significantly, with some models even refusing to provide directions in certain cases. This highlights a critical gap in capabilities among smaller models, particularly regarding logical route navigation and local knowledge. The findings underline important implications for the AI/ML community, particularly in understanding how LLMs utilize internal memory structures to generate navigation instructions. The results suggest that while models like Opus 5.5 demonstrate good memory of global positioning, they often lack an effective representation of road connectivity. This indicates a future need for improved training methods and integration of real-time mapping data, especially as reliance on AI for practical applications like navigation increases. These insights encourage further exploration into how LLMs can better synthesize and apply geographic data to provide more accurate and reliable directional assistance.