Can a MUD evaluate LLMs? A $99 proof of concept (cruciblebench.ai)

CrucibleBench has introduced an innovative proof-of-concept for evaluating large language models (LLMs) using a multi-user dungeon (MUD). This unique approach leverages the constraints of text-based environments from the early internet, such as limited command options and persistent game states, to measure model behavior in socially complex scenarios. Unlike static benchmarks that assess knowledge in isolation, CrucibleBench focuses on dynamics like trust, relationship management, and action efficiency, allowing for detailed analysis of how models earn trust and engage with their environment. The significance of this method lies in its ability to uncover measurable failure modes that traditional benchmarks overlook. For instance, it reveals issues like dialogue looping and wrong-room interactions, demonstrating that many LLMs struggle to adapt conversational strategies or manage state effectively. Early results indicate that the inclusion of an LLM judge significantly alters rankings, highlighting the necessity for per-model agreement and ranking stability in evaluation. With a full release including 650 transcripts and scoring code, CrucibleBench aims to foster further development and calibration of this novel benchmark, providing the AI/ML community with a powerful tool for assessing LLM performance in more nuanced and realistic settings.