Anthropic's Argument for Mythos SWE-bench improvement contains a fatal error (www.philosophicalhacker.com)

Anthropic has raised concerns about the validity of Mythos' performance claims on the SWE-bench, suggesting that their results may be skewed by issues of memorization. In their analysis, Anthropic demonstrated that while Mythos scores higher on SWE-bench compared to Opus 4.6, the use of a probabilistic memorization detector could lead to misleading conclusions. For instance, when including results deemed memorized with lower confidence, Mythos appears to achieve a 92% success rate, compared to Opus 4.6’s 82%. However, Anthropic acknowledges that their detection methods are imperfect, raising questions about the reliability of the purported gains. The implications of this analysis are significant for the AI/ML community, as it highlights potential flaws in benchmarking practices that rely on memorization detection. The critique underscores the need for more rigorous evaluation metrics to assess model performance accurately. A simple Python simulation provided in the analysis illustrates how even an imperfect detector could falsely indicate genuine improvements in model efficacy. Until the accuracy of these detection methods is more thoroughly quantified, claims regarding Mythos’ superiority on SWE-bench remain questionable, emphasizing the importance of transparency in AI performance metrics.