Chess LLM Benchmark: Evaluating LLMs' ability to play chess (github.com)

A new benchmark has been developed to evaluate the chess-playing capabilities of large language models (LLMs) by allowing them to compete against established chess engines and each other. Utilizing the Glicko-2 rating system, LLMs receive a rating comparable to Lichess Classical ratings based on their performance against calibrated "anchor" chess engines, such as Stockfish and Maia. The benchmark mandates that LLMs make legal moves per chess rules; illegal moves result in a warning, and a second infraction leads to forfeiture. Key features include a comprehensive leaderboard, game library, and client-side analysis tools, making it a valuable resource for assessing LLMs in competitive environments. This initiative is significant for the AI/ML community as it creates a structured framework to understand LLM proficiency in strategic reasoning tasks, like chess. With the ability to test multiple models simultaneously, the benchmark can reveal differences in reasoning abilities and efficiency, aiding in the development of more advanced AI. As LLMs have become increasingly capable across diverse domains, this benchmark not only contributes to the field of AI but also enhances our understanding of machine cognition in complex situations, potentially paving the way for applications in various strategic domains beyond chess.