CursorBench 3.1 (cursor.com)

CursorBench has released its latest version, CursorBench 3.1, enhancing benchmarks for AI-driven code modeling by introducing new tasks that emphasize codebase understanding, bugfinding, planning, and code review. The latest rankings feature models like Fable 5 Max leading in performance with a score of 72.9%, while also revealing the costs associated with each model's execution. The pricing for these models varies, with Fable 5 Max priced at $18.02 per million tokens, reflecting a significant resource investment alongside performance metrics, crucial for developers and researchers seeking efficiency. This update is important for the AI/ML community as it refines the methods for assessing and comparing AI models' capabilities in programming contexts, addressing real-world challenges faced by developers. Enhanced grading criteria for specific tasks aim to improve the evaluation process of model performance. The inclusion of diverse tasks ensures that benchmarks remain relevant to evolving coding practices, pushing forward the development of smarter coding assistants and ultimately aiding in the broader application of AI in software engineering.