Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases (withspecific.com)

Real-SWE, a new benchmark tool, has been introduced for evaluating AI models against private, real-world enterprise codebases. By licensing actual code from a company, this benchmark presents engineering tasks that reflect the complexity and context of ongoing business operations, such as managing billing and tax compliance. The evaluation reveals significant challenges for AI coding agents, with top models like Fable 5.1 achieving a resolution rate of only 38.8%, underscoring the difficulty of addressing company-specific coding conventions and the need to integrate seamlessly with existing infrastructure. The significance of Real-SWE lies in its approach to bridge the gap between theoretical AI capabilities and practical software engineering requirements. By focusing on genuine tasks that real software engineers encounter, the benchmark highlights the limitations of current models in handling nuanced, enterprise-level coding challenges. Overall, Real-SWE emphasizes the importance of contextual understanding in AI, revealing that while progress has been made, AI still struggles significantly when tasked with real-world coding scenarios that demand high standards of accuracy and efficiency.