Agents against broken ML workloads (zhebrak.io)

A new benchmarking study titled "Agents against broken ML workloads" evaluated the performance of various AI agents in fixing malfunctioning machine learning (ML) jobs across different access levels. Conducted on a two-node Slurm cluster utilizing PyTorch and Docker, the study simulated real-world ML failures, such as shape mismatches and out-of-memory errors. The agents were tested under three access scenarios: configuration files and code only, real-time logs, and full root access. Results revealed that providing higher access significantly improved job-fixing efficiency, with performance enhancements of up to eight times noted. Notably, GPT-6 Luna emerged as the most cost-effective model for this task, achieving impressive price-performance metrics compared to other agents. The findings underscore the importance of access levels in troubleshooting ML workloads, suggesting that stronger models are critical when access is limited to configuration files. Moreover, live logs can often serve as a viable substitute for full cluster access, hinting at potential future applications in real-world scenarios where not all data may be available. However, the study's limitations include its focus on CPU usage and potential biases introduced by the coding agents, which raises questions about the generalizability of the results to GPU clusters. The research highlights a growing need for adaptive agent frameworks capable of efficiently diagnosing and resolving ML operational issues in varied environments.