Mass attempted replication of old findings of LLMs with new models (zou-group.github.io)

A recent study revisited 50 well-cited claims about large language models (LLMs), revealing that a significant number of findings may have a short lifespan. The analysis, which encompassed 20 models released between July 2024 and September 2026, found that 44% of the claims tested had expired entirely, with an additional 34% showing significant weakening. This highlights a critical issue in the AI/ML field: the rapidly changing landscape of LLM capabilities and the necessity for continual validation of existing research. The study involved a broad range of claims, including models' reasoning abilities, biases, and susceptibility to manipulation. It noted that claims related to the models' capacities, such as arithmetic and reasoning, were among those that had faded. The research methodology included comparing performance trends over time to assess how and when claims deteriorate. Notably, only a small percentage of claims—16%—remained robust, with some even strengthening under new model iterations. This underscores the importance of ongoing scrutiny and adaptation in AI research, as developers and practitioners must consider that findings might quickly become outdated as models evolve.