People who use ChatGPT for writing are accurate detectors of AI text (2025) (arxiv.org)

A recent study published in the field of computation and language reveals that individuals who frequently utilize ChatGPT for writing tasks possess an impressive ability to identify AI-generated text produced by leading models like GPT-4o and Claude. By analyzing 300 non-fiction English articles, researchers found that these "expert" annotators achieved a remarkable accuracy rate, misclassifying only one article among the 300. This performance significantly surpassed that of existing commercial and open-source detection tools, even against sophisticated evasion tactics such as paraphrasing. The significance of this finding lies in the implications for both human and automated AI text detection systems. The annotators were able to leverage a combination of lexical indicators and higher-order textual characteristics, such as formality and originality, which current automated detectors struggle to interpret accurately. This study not only highlights the innate skills developed through regular interaction with AI tools but also offers valuable annotated datasets and code to foster ongoing research in the domain, ultimately enhancing the landscape of AI detection technologies in the future.