FrontierCode: Can AI models write code maintainers would merge? (artificialanalysis.ai)

Cognition has launched FrontierCode 1.1, a benchmark designed to evaluate whether AI models can make code changes that maintainer teams of real open-source repositories would consider acceptable for merging. The benchmark measures not just the success of tests but also the quality of code changes, focusing on issues such as excessive modifications, altering irrelevant files, or submitting ineffective tests. Current leading models perform below expectations, with a failure rate exceeding 40%, underscoring that while AI models are increasingly proficient at coding tasks, their outputs often require significant refinement. This benchmark's establishment is significant for the AI/ML community as it provides a rigorous testing standard that addresses real-world coding complexities involved in collaborative software development. In its latest evaluation, Claude Opus 5.5 emerged as the top performer, achieving a 52% score with a pass rate of 58%. Leading models scored between 43% and 52%, revealing a tightly packed leaderboard and highlighting the ongoing challenges in programming by AI. By emphasizing the importance of maintaining high coding standards, FrontierCode 1.1 serves as a critical tool for assessing and guiding future developments in coding capabilities of AI models, fostering a deeper integration of AI tools in software engineering processes.