How we measured AI writing across arXiv, and where the measurement breaks (unslop.run)

A recent analysis of 12,750 arXiv papers reveals that approximately 32% of submissions are now flagged as AI-written, a significant rise observed following the introduction of large language models like ChatGPT. The study employed a calibrated detector specifically designed for academic writing, boasting a low false-positive rate of 0.4%. This methodology allows for an accurate comparison with pre-LLM submissions, supporting claims that the increase in AI-like writing is not simply attributable to detector sensitivity. Notably, the rise in flagged papers is most pronounced in fields such as computer science (65%) and quantitative biology (56.3%), while mathematics shows the lowest rate at 0.7%. This finding holds crucial implications for the AI/ML community, indicating a substantial shift in authorship dynamics and the potential for AI tools to influence academic publishing standards. As the prevalence of AI-generated text grows, discussions about originality, authorship, and the integrity of research outputs will intensify. The study also highlights limitations, such as variations in detector sensitivity across fields and the challenges in interpreting low scores, particularly in mathematical papers that may not align with the detector's training. Overall, this trend raises important questions about how AI technology is reshaping the landscape of academic writing and the necessity for ongoing scrutiny of AI's role in research.