We gave open models root on broken servers and graded them on the machine (glassmkr.com)

In a novel testing exercise, the team behind the monitoring product Glassmkr evaluated various open-weight AI models to assess their real-world operational capabilities by giving them root access to intentionally problematic servers. Unlike conventional assessments that often rely on model-produced summaries, this method utilized a read-only API key that only validated fixes based on changes made directly to the server. The study encompassed nine models with parameters ranging from 8B to 120B, testing their ability to resolve specific server issues like unsynced clocks and firewall configurations. The results revealed that model effectiveness was not strictly correlated to size; smaller models often outperformed larger ones in specific scenarios, challenging the notion that bigger parameters equate to better performance. Notably, the interaction with the AI also significantly influenced outcomes; models displayed vastly different capabilities depending on whether they were instructed via plain-text or function-call methods. This insight underscores the importance of aligning model training with evaluation frameworks, highlighting that the effectiveness of an AI model in practical applications may depend more on the context of its use than on its size or apparent complexity.