Zork-bench: An LLM reasoning eval based on text adventure games (www.lowimpactfruit.com)

The recent launch of "zork-bench," a novel evaluation tool for large language models (LLMs), is shaking up the AI/ML community by leveraging the classic text adventure game Zork to assess reasoning and problem-solving capabilities. Developed by a team at the Recurse Center, zork-bench provides a structured environment where LLMs interact with Zork's complex puzzles, revealing that many modern models struggle significantly, often scoring less than humans unfamiliar with the game. This observation raises intriguing questions about LLMs' reasoning processes, particularly as they are expected to excel in tasks grounded in well-documented contexts like Zork. Key technical components of zork-bench include a harness that allows LLMs to maintain a map and manage items, simulating the decision-making required to progress in the game. The project aims to analyze LLM behavior, such as planning effectiveness and memorization tendencies, by tracking actions taken across various scenarios. Initial results indicate that LLMs often expend excessive effort without substantial progress, highlighting a disconnect between their training on vast datasets and their performance in structured problem-solving contexts like text adventures. This work not only enhances understanding of LLM limitations but also opens avenues for further exploration in AI reasoning capabilities using interactive narratives.