Agents on Rails: Best model solves 35% of feature benchmark runs (rubyonrails.org)

In a significant advancement for the AI/ML community, the Agents on Rails project has unveiled Stage 2 of its benchmark series, challenging AI models to handle more complex, real-world feature tasks. The standout performer, GPT-6 Astra, successfully solved 35% of the benchmark runs, demonstrating a combination of accuracy, cost-effectiveness, and speed, achieving a median completion time of 9 minutes per run with a campaign cost of $150.47. This stage differs from its predecessor by requiring models to respond to fully real-world ticket scenarios, mirroring how developers operate in practice, rather than focusing solely on technical knowledge. The results highlight critical implications for AI-driven development tools, showcasing that current models struggle with comprehensive task completion, particularly when edge cases are not explicitly outlined. While models like Luna excelled in simple tasks, they faltered significantly in this round, failing to complete any of the feature tickets due to their inability to engage in necessary planning and detail-oriented work. As the project progresses toward Stage 3, which will include even more demanding criteria, the benchmarks are set to push the boundaries of what AI agents can accomplish in software development, ultimately aiming to build capability that extends beyond single tasks to complex projects.