AI coding models state their assumptions only 46% of the time (bito.ai)

A recent study analyzing 29 AI coding models against 60 real engineering tasks reveals that these models state their assumptions only 46% of the time. This research is pivotal for engineering teams deciding which AI tools to adopt, as it highlights significant discrepancies between price and performance. Notably, the top-performing model, claude-opus-5, achieved a score of 54.5 for $135.51, while other models, including the open-weight deepseek-v4.1-flash, scored 50.5 at just $2.67, underscoring the potential for cost-effective solutions that deliver comparable results. The study also indicates that traditional pricing metrics (like cost per token) no longer correlate reliably with actual costs, as models with higher list prices can end up being cheaper in practice, depending on their efficiency in task execution. This shifts the focus to the cost per successful task completed. Moreover, how a model performs when faced with vague instructions greatly varies, impacting its practical utility in real-world settings. With a crowded landscape of AI options, this research arms decision-makers with essential insights to navigate the evolving market and optimize their engineering processes efficiently.