Do AI Detectors Work Well Enough to Trust? (www.chicagobooth.edu)

Recent research from Chicago Booth highlights the effectiveness of AI detectors designed to differentiate between human-written and AI-generated text. As generative AI's capabilities grow, there is increasing concern over its potential misuse in various fields, prompting a surge in demand for reliable detection tools. The study tested three commercial detectors—Pangram, Originality.ai, and GPTZero—and one open-source tool, RoBERTa, across a range of text lengths and formats. Results indicate that all three commercial tools maintain low false positive rates (below 1%) and show a relatively high level of accuracy in longer pieces, although performance declines with shorter texts. The findings present significant implications for educational institutions and employers, as they face the challenge of using these detectors to minimize misuse while avoiding false accusations against individuals. The researchers propose a "policy cap" framework to help organizations set acceptable thresholds for false positives. Notably, Pangram emerged as the most reliable tool, effectively maintaining high accuracy levels across formats. However, the evolving nature of AI detection tools suggests that ongoing performance audits will be crucial for ensuring their fair and effective application in real-world settings.