Why Are Large Language Models So Terrible at Video Games? (spectrum.ieee.org)

Recent discussions highlight the significant challenges that large language models (LLMs) face in playing video games, despite their rapid advancements in other areas like coding. While some LLMs, such as Gemini 2.5 Pro, have demonstrated success in beating certain games like Pokémon Blue, they still lag behind human players in terms of speed and performance quality, often requiring specialized software for assistance. Julian Togelius from New York University’s Game Innovation Lab emphasizes that the difficulties LLMs encounter stem from the diverse mechanics and input structures of different games, making it challenging to create a general AI capable of mastering all gaming types. This limitation reflects a broader issue in the field of AI, as the lack of progress in video game proficiency indicates that LLMs and similar models do not possess the generalized understanding necessary for varying tasks outside their training environments. The differences between structured tasks, like coding, and the chaotic and less predictable dynamics of video games contribute to these shortcomings. Togelius points out that while specialized AIs like Google’s AlphaZero can excel in specific games like chess and Go, they require significant retraining and adaptation. The discrepancy illustrates the need for better benchmarks and understanding of AI learning processes, suggesting a focus shift towards developing more flexible models capable of navigating the intricate landscapes of video games.