How I made a benchmark mined from my own repo that agreed with FrontierCode (www.stet.sh)

A researcher has developed a unique benchmark for evaluating AI models using code from their own repository, achieving a remarkable correlation of 0.96 with Cognition's FrontierCode, which is known for its comprehensive performance metrics. This benchmark, generated from 24 tasks mined directly from the author’s Git history, highlights the importance of context and cost-effectiveness in assessing machine learning models. While FrontierCode uses 150 tasks requiring extensive manual effort, this new benchmark automates the evaluation process, exposing the discrepancies in cost between different environments. For instance, the author found that certain models were 3.5 to 5 times more expensive on their setup compared to FrontierCode, emphasizing that effective evaluation must consider the nuances of individual codebases. This work is significant for the AI/ML community as it underscores the need for tailored evaluation metrics that reflect real-world usage, rather than relying solely on public benchmarks that may not accurately capture specific conditions. By automating and customizing the evaluation process—taking into account factors like task appropriateness, model configurations, and infrastructure reliability—the author aims to provide engineers with reliable tools to navigate the complex landscape of AI model selection, helping them optimize for quality, cost, and performance specific to their unique environments.