DeepCodeBench: Real-World Codebase Understanding by Q&A Benchmarking (www.qodo.ai)

Qodo has introduced DeepCodeBench, a novel benchmark dataset designed to evaluate AI systems on real-world codebase understanding through question-answering tasks. Unlike prior benchmarks that rely on isolated or synthetic code snippets, DeepCodeBench leverages realistic developer questions generated from pull requests spanning multiple files and interconnected code blocks in large repositories. This approach better mirrors the complexity developers face when navigating enterprise-scale codebases, offering more authentic challenges for retrieval-augmented AI models. The dataset comprises 1,144 curated question-answer pairs derived from eight open-source projects, with questions crafted using large language models prompted on contextual code and PR metadata. These questions range from deep, focused inquiries on specific code behavior to broad, architectural-level questions requiring multi-file retrieval. Evaluation employs an objective "fact recall" method, verifying discrete facts extracted from ground-truth answers against model responses, ensuring scalable and rigorous performance measurement. Baselines show Qodo’s own DeepResearch agent leading with roughly 76% fact recall—outperforming OpenAI Codex and other competitors—highlighting strong reasoning and semantic search capabilities essential for real-world code understanding. By releasing the dataset, prompts, and evaluation methodology, DeepCodeBench fills a critical gap for benchmarking retrieval and comprehension in complex codebases. It provides the AI/ML community with a practical, challenging foundation to advance code intelligence systems capable of supporting developers in navigating large-scale, evolving software projects.