Researchers Use D&D to Test AI's Long-term Decision-making Abilities (today.ucsd.edu)

Researchers at the University of California San Diego have introduced an innovative approach to evaluate the long-term decision-making abilities of Large Language Models (LLMs) by simulating gameplay in Dungeons & Dragons (D&D). This method utilizes the game’s complex rules, collaborative nature, and extended play sessions to provide a robust testing environment for AI agents that operate autonomously over longer periods. By requiring the LLMs to engage in role-playing, adhere to game rules, and interact with both human and AI players, researchers aim to address the current lack of benchmarks for assessing LLM performance in long-term tasks. During the study, three LLMs were tested in various D&D scenarios, revealing their ability to maintain character consistency and strategize collaboratively in real-time. Claude 3.5 Haiku emerged as the most effective model, with GPT-4 closely following, while DeepSeek-V3 lagged behind. Notably, the models displayed unexpected behaviors, such as personifying game elements and engaging in dramatics during battles. Future research will expand beyond combat scenarios to simulate entire D&D campaigns and explore applications in negotiation and strategic planning, pushing the boundaries of what AI can achieve within complex, interactive environments.