cua-speedrun: Standardized Benchmarking of the Speed of Computer-Use Agents (cuaspeedrun.com)

The newly announced cua-speedrun benchmark introduces a standardized system for evaluating the speed and efficiency of computer-use agents by measuring not just task completion success but also the time taken and computational costs involved. By utilizing a pay-per-second cloud service on Modal and maintaining a consistent virtual machine environment, this benchmarking tool enables fair comparisons across different models and tasks. It supports a range of applications and tasks, allowing researchers to submit agents that can operate within fixed constraints, ensuring reliable results. This framework is significant for the AI/ML community as it addresses a gap in benchmark assessments by focusing on time and resource efficiency rather than simply success rates. The cua-speedrun specifically features a unique fast I/O mode designed to reduce the lag between actions, providing deeper insights into how latency affects agent performance. With a correlation of 0.98 between reduced task sets and full benchmarks, the system promises a practical and efficient means to evaluate AI agents, fostering advancements in agent development and optimization.