AI agents write Ruby but can't navigate it: a 5-model, 13-codebase benchmark (github.com)

A recent benchmark has revealed that while AI agents can generate Ruby code effectively, they struggle to navigate complex codebases to identify dependencies essential for refactoring. This study evaluated five AI models across 13 real-world Ruby codebases, focusing on their ability to find all references to a central model before changes are made. Results showed a stark difference in performance: agents performed adequately with smaller, simpler gems but fell short in larger applications, often missing critical dependencies. However, integrating a structural map significantly improved performance, with the best model (Claude Opus 4.8) achieving a notable increase in recall for dependents. The findings are crucial for the AI/ML community as they highlight the limitations of AI in understanding and navigating intricate code relationships, which are common in real-world applications. The introduction of a code structural map allowed agents to better interpret complex dependencies, demonstrating that enhancing AI tools with comprehensive code insight can markedly improve their effectiveness. This study underscores the importance of context and relationships in software development and suggests that combining AI models with augmented tools can mitigate the risk of errors during code changes, ultimately leading to safer deployments and more robust code management practices.