LLM Chess – Leaderboard (maxim-saplin.github.io)

Researchers have launched an innovative leaderboard to evaluate the chess capabilities of Large Language Models (LLMs) in a multi-turn dialogue setting. In this setup, LLMs engage in chess games against a Random Player or the Komodo Dragon Chess Engine, analyzing their ability to follow instructions and make strategic moves. Starting from a baseline in 2024 where many models struggled with instruction adherence and basic gameplay, the new leaderboard has introduced advancements that show significant improvements in chess skills and instruction-following, particularly with more sophisticated reasoning models introduced in 2025. This initiative is significant for the AI/ML community as it provides a structured framework to assess LLMs' reasoning and game-playing capabilities, anchoring their performance to a recognized Elo rating system. The leaderboard quantifies metrics such as player efficiency, game duration, and cost per Elo, offering insights into the interplay between model complexity and performance. As LLMs continue to evolve in their chess play, this research signifies a critical step toward understanding the deeper cognitive abilities of AI agents, paving the way for further innovations in AI reasoning and decision-making processes.