2,218 Gary Marcus AI claims scored against evidence (dataset) (github.com)

A comprehensive analysis has been conducted on 2,218 claims made by noted AI skeptic Gary Marcus, utilizing two advanced language models, Claude Code (Opus 4.6) and Codex (ChatGPT). The study revealed that 59.9% of Marcus's claims were supported by evidence, while 33.7% were classified as mixed and 6.4% contradicted. Notably, Marcus's critiques regarding specific technical issues, such as LLM security vulnerabilities, received robust endorsement from the evidence—showing a complete support rate. Conversely, his predictions about the AI market were less reliable, with only 27% of claims regarding a potential "GenAI bubble" being supported. This analysis is significant for the AI/ML community as it provides a nuanced view of Marcus's claims, highlighting areas of validity while also addressing misconceptions. The methodology employed, using a hybrid approach between two LLM pipelines, enhances the rigor of the evaluation process. The presence of both supporting evidence and contradictions in Marcus's work underscores the complexities of the AI landscape, suggesting a need for critical engagement with prevailing narratives and predictions in the field. The findings also emphasize the value of leveraging AI tools to assess claims about AI, potentially fostering a more grounded understanding of the technology's current and future state.