I built a benchmark for testing LLMs playing Gomoku (github.com)

A new tool called GomokuBench has been launched to benchmark large language models (LLMs) against a classical search algorithm using the game Gomoku. This lightweight framework is designed for researchers and AI developers to assess whether general-purpose language models can outperform a deterministic AlphaBeta search engine in a well-defined board game environment. It offers features such as dual mode play between two LLMs, detailed move logs, and structured JSON results for easier analysis and reproducibility. Currently, no LLM has succeeded in defeating the built-in AlphaBeta engine, with all tested models reporting a score of 0-10. The significance of GomokuBench lies in its ability to provide a simple yet adversarial platform for comparing AI models, thus facilitating rapid experimentation and benchmarking. The clear rules of Gomoku allow for straightforward interpretation of model performance, making it an ideal scenario for exploring AI reasoning and decision-making capabilities. Additionally, the benchmark supports easy integration with model APIs and command-line execution, enhancing usability for varied AI research and development purposes. As AI continues to evolve, tools like GomokuBench play a crucial role in evaluating the effectiveness of LLMs against traditional algorithms, potentially guiding future advancements in the field.