A Good College Essay but a Bad Police Report: A triple blind study (www.crimrxiv.com)

Recent research evaluating AI-powered report-writing tools in policing has unveiled concerning insights about their effectiveness. Conducted as a triple-blind study involving 92 expert law enforcement reviewers, the experiment assessed 80 reports—40 generated with AI assistance from Axon's Draft One and 40 without. While AI reports were found to be more complex and less readable, reviewer assessments indicated no significant quality improvements; in fact, AI-generated reports were rated significantly lower in accuracy. This challenges the prevailing market claims that AI can elevate report quality, highlighting a disconnect between surface-level improvements in writing style and the substantive skills required for effective police reporting. The implications for the AI/ML community are substantial. The study underscores the necessity for a reevaluation of how AI tools are integrated into professional settings, emphasizing that technology cannot replace the critical judgment and contextual expertise inherent in tasks like police report writing. As police departments increasingly adopt AI solutions to alleviate administrative burdens, it’s vital that governance frameworks pivot towards rigorous audits and disclosure mechanisms rather than relying solely on supervisory perceptions of quality. Without such changes, the adoption of AI in policing could perpetuate existing quality issues rather than resolving them.