Land or Water? (twitter.com)

Andrej Karpathy recently showcased an interesting evaluation technique using large language models (LLMs) to determine whether given latitude and longitude coordinates correspond to land or water. By querying the model 16,200 times, he was able to generate a plot visualizing its performance. Notably, a model that answered "water" for every query achieved about a 70% accuracy, highlighting the challenge posed by Earth’s vast oceans. In comparison, the Sonnet 4.5 model, performing at 60%, surprisingly underperformed, demonstrating that a baseline level of geographical knowledge is essential for nuanced understanding. This experiment emphasizes the potential of LLMs to compress and retain vast geographical knowledge from the internet, revealing their capabilities to discern geographical features akin to high-performing models from just a year ago. The findings not only underscore the relevance of model architecture and cost-effectiveness in training but also open up exciting avenues for generating labeled maps of continents and countries. Such advancements could significantly enhance applications in geography, environmental science, and beyond, providing a richer context for AI understanding of the world.