Coding Agent Is a Slot Machine (blog.kvit.app)

A recent experiment comparing the coding agents Claude Sonnet 4.5 and Vibe (Devstral 2) highlights significant inconsistencies in the performance of AI models, even when tackling the same tasks. Testing on a subset of 45 GitHub issues, the researcher executed 10 runs per agent per case, resulting in roughly 40% of test cases yielding different outcomes despite identical conditions. This variability raises important questions about how reliable benchmark scores reflect real-world performance; while Claude had a pass rate of 39.8% and Vibe 37.6%, the consistency of solutions varied widely, with patch sizes swinging dramatically even for cases deemed successfully resolved. This analysis underscores a crucial point for the AI/ML community: the need to prioritize consistency over sheer performance metrics. As agents are refined, the capacity to deliver reliable and predictable outputs is paramount. The findings suggest that a model that consistently solves a reasonable percentage of tasks—despite a potentially lower overall success rate—may be more practical for developers than a faster or higher-scoring model that frequently produces erratic results. The researcher aims to delve further into local model testing, exploring how quantization and other factors might influence both performance and variability.