Better Models: Worse Tools (lucumr.pocoo.org)

Recent tests have revealed a troubling issue with newer Claude AI models, particularly Opus 4.8 and Sonnet 5, which are struggling to correctly format tool calls for file edits by including extraneous, unrecognized fields. This dysfunction leads to frequent rejections of valid requests as the models fail to adhere to the expected schema. Surprisingly, this is a regression compared to older models in the same family, which used to handle tool calls more robustly. The issue appears to stem from the post-training reinforcement of models using an overly forgiving tool schema from Claude Code, resulting in a lack of performance in adapting to different tool formats. The implications of this regression are significant for the AI/ML community, highlighting that model training and post-training environments can dramatically influence the adaptability and reliability of AI systems. The reliance on a specific tool schema means that these models might become less effective in handling diverse real-world applications where different formats are needed. This situation raises concerns about how tool schemas impact AI performance, suggesting that a failure to account for varied structures could lead to substantial limitations in functionality and user experience.