Open Codenames: a new way to benchmark LLMs (github.com)

Open Codenames has been introduced as an innovative platform for benchmarking large language models (LLMs) through a game of Codenames, where players interact with AI teammates and opponents. In this version, users can provide their own LLM keys from various providers like OpenAI and Google, while ensuring that their keys remain secure and private. The game utilizes a built-in picture deck featuring public-domain engravings, allowing for a unique blend of visual and linguistic reasoning as AIs generate clues and engage in gameplay. The desktop and browser versions of the game are built on technologies such as Electron, TypeScript, and React, and offer detailed player documentation to facilitate engagement and development. This initiative is significant for the AI/ML community as it not only provides a fun and interactive way to evaluate LLM performance but also emphasizes the importance of reasoning in AI interactions. With features like a robust debrief after each game and the ability to dictate clue strategies based on LLM evaluations, researchers and developers can gain insights into how different models interpret information and generate responses. Additionally, the open-source nature of the project encourages collaboration and innovation within the community, fostering improvements in AI reasoning and interaction capabilities.