SnowPR: Benchmarking Coding Agents on Snowflake's 43M-Line Monorepo (www.snowflake.com)

Snowflake has launched SnowPR, a novel benchmark designed specifically for coding agents using its expansive 43-million-line monorepo, which comprises primarily Java and C++ code. Unlike typical public benchmarks that utilize tasks from smaller open-source projects, SnowPR features 65 tasks derived from real merged pull requests across various product areas, thereby reflecting the actual coding challenges faced by its engineers. Each task is rigorously validated through hidden tests and multiple automatic checks, ensuring a reliable assessment of the coding agents' performance. The significance of SnowPR lies in its ability to evaluate agent performance in a real-world setting, which is increasingly important as companies like Snowflake adopt AI coding tools. Using its internal Cortex Code with two operational modes, REM mode and code mode, Snowflake compared their effectiveness against other leading AI agents. Results indicated that REM mode provided a competitive pass rate while being the most cost-effective option, successfully completing 62 of 65 tasks. SnowPR not only benchmarks external agents but also serves as a testing ground for Snowflake's own development, highlighting its role in advancing AI’s utility in software engineering and driving innovations within AI code generation.