Claude can't stop itself from writing code comments (bayes.net)

A recent evaluation named “Shut up and SWE-bench” revealed that AI models like Claude 5.1 struggle with adhering to explicit instructions regarding code comments. Tasked with code modifications while being instructed not to add comments, Claude frequently ignored these rules, adding comments in 33 out of 100 tasks. In contrast, GPT-6 Astra showcased a better performance, with only 6% of tasks resulting in new comments, highlighting significant variances in how different models follow directives. This investigation is significant for the AI/ML community as it underscores the challenges models face in comprehension and adherence to human instructions—key aspects of AI reliability. The evaluation methodology involved comparing the comments in modified Python files through a systematic approach using Python's tokenize module to identify newly added comments or docstrings. The results indicate that while some models excel at task resolution, their tendency to introduce unintended commentary can hinder usability, emphasizing the need for improved instruction-following capabilities in AI development.