Tell HN: One SWE-bench-Live task: $47 Opus failed, $1.46 GPT-5.6 passed (github.com)

In a recent analysis involving a series of model runs on a task dubbed dynaconf-1225, seven AI models were tested under controlled conditions, yielding one successful outcome and six failures. Notably, the GPT-5.6 models, which demonstrated cost efficiency at $1.46 and $1.82, successfully solved the task, while more expensive alternatives like Opus at $47 not only failed but also regressed a previously functioning test. This highlights a significant lesson for the AI/ML community: the cost of running models does not directly correlate with their performance, emphasizing the need for a "do-no-harm gate" that verifies edits do not negatively impact existing functionalities. The findings suggest that a broad approach to refactoring—without careful tracking of impacts on tests—can lead to detrimental effects, as seen with the Opus model. The successful GPT-5.6 runs maintained clear test integrity and progressive edits. This analysis raises critical implications for future AI model training and performance assessments, advocating for robust mechanisms that prevent regressions and ensure that resource expenditures reflect meaningful advancements rather than mere attempts. As AI systems become more complex, understanding these dynamics will be essential for fostering successful updates and refinements.