DishwasherBench – Can Frontier Models Stack a Dishwasher? (github.com)

DishwasherBench is a novel benchmarking tool that simulates loading a Bosch dishwasher, designed to evaluate the performance of language models in a gamified environment. Players, both human and AI, engage in a game called Spotless, where they load a 3D model of the dishwasher, and the resulting wash outcome is scored based on cleanliness. The current version, 0.1.0, introduces deterministic loading mechanics and a baseline wash simulation using the Eco 50 program, providing points for optimal item placement. As the game progresses, players must tackle increasingly complex loading scenarios, requiring strategic decisions to maximize cleanliness. This initiative is significant for the AI/ML community as it combines physical simulation with AI training, pushing the boundaries of language models’ task capabilities. The benchmarking also highlights key technical implications, such as the need for accurate scoring mechanisms for different dishware types, which currently focus on simpler parameters like ensuring items are placed correctly. The project employs rigorous testing against a real washing machine, validating its scoring against known benchmarks, while offering various modes of interaction for both casual players and AI agents via a web interface. As the project develops, it aims to refine its scoring system and expand its testing across multiple platforms.