The Opus 4.7 reasoning curve - Medium is the best default? (www.stet.sh)

Opus 4.7's recent evaluation highlights a surprising finding: when tested across various reasoning effort levels (low to max) on 29 tasks from the GraphQL-go-tools repository, the medium setting yielded the best performance. Contrary to traditional assumptions that increased reasoning correlates with enhanced intelligence, the model exhibited a non-monotonic reasoning curve, peaking at medium. This means that while lower settings are faster and cheaper, they sacrifice correctness, whereas higher settings did not deliver proportional improvements. For instance, examples like PR #1260 demonstrated that medium not only recovered effectively into functional patches but also maintained a better correspondence with human-authored changes compared to high and xhigh settings, which often concluded tasks with "no changes needed" despite having more reasoning resources. This finding is particularly significant for the AI/ML community as it calls into question the conventional wisdom regarding model reasoning capabilities, suggesting that overly complex reasoning might hinder performance rather than enhance it. The implications are clear: developers should focus on optimizing configurations for AI systems based on empirical evidence rather than assumptions, emphasizing a balance between effort and effectiveness. The study encourages further exploration into adaptive reasoning within generative models, proposing that agents could benefit from refining their self-identified reasoning strategies on real-world tasks rather than merely increasing reasoning budgets.