Cto.new launches new benchmark using real end-to-end coding tasks (cto.new)

A new online coding benchmark called CTO Bench has been launched, aimed at evaluating large language models (LLMs) based on their performance in real-world coding tasks. This platform measures the percentage of merged code from completed tasks, offering a leaderboard that tracks a 72-hour rolling success rate with a two-day lag to account for task resolutions. Only models that meet a minimum usage threshold are included, ensuring that the results reflect statistically significant data from active users of the cto.new tool. CTO Bench is significant for the AI/ML community as it provides a standardized method for assessing the coding capabilities of LLMs, promoting transparency and competition among models. The integrated toolset, which includes functionalities like file reading and writing, regex searching, and shell command execution, allows developers to test models in a real coding environment effectively. By fostering a community-driven approach to benchmarking, CTO Bench aims to enhance model development and improve the overall quality of AI coding assistants.