I turned my game into an eval (crux.lakin.dev)

Dustin Lakin has transformed his rogue-like RPG game, "The Crux," into a platform for evaluating AI reasoning capabilities, rather than relying on reinforcement learning. This innovative approach allows for a deeper analysis of how different AI models tackle game mechanics and decision-making under consistent conditions. The game simulates various player behaviors by integrating deterministic randomization and extensive observability, offering a rich testbed for artificial intelligence exploration. By conducting simulations over 1,000 seeded runs, Lakin aims to evaluate how effectively AI can craft character builds using specific talents, behaviors, and tactics. This project is significant for the AI/ML community as it shifts the focus from traditional reinforcement learning metrics to a nuanced understanding of AI reasoning and adaptation. Lakin’s methodology provides insights into the iterative learning and memory capabilities of models as they engage with complex systems, offering a fresh perspective on AI performance across different runs. Additionally, it highlights the importance of detailed logging and analysis of runs to track AI improvement over time, creating a valuable benchmark for future AI-related developments in game design and beyond.