GoBench: Evaluating LLMs on 9×9 Go using KataGo opponents as Elo anchors (rolandgao.com)

GoBench has been launched to evaluate the performance of leading large language models (LLMs) in the game of 9×9 Go, using a series of calibrated KataGo opponents as Elo anchors. This benchmarking tool is significant for the AI/ML community as it aims to address the observed inconsistency in language models, showcasing high capabilities in certain domains like math and coding while underperforming in others. A key goal of GoBench is to assess the general reasoning abilities and context-based continual learning of these models, which is vital for advancing towards artificial general intelligence (AGI). The benchmarking involves two tracks: one for general reasoning using multi-turn API interactions, and another that employs a setup mimicking continual learning by varying exposure times to training data before evaluations. The models' performance is quantified through Elo ratings, providing a nuanced insight into their strategic reasoning and learning adaptability within a competitive environment. Researchers are encouraged to extend GoBench's framework to further explore continual learning capabilities, positioning it as a crucial resource for advancing understanding and performance of AI systems in complex tasks.