High school’s AI security system confuses Doritos bag for a possible firearm (techcrunch.com)

A Baltimore County high school student was handcuffed and searched after an AI-powered gun-detection system flagged a Doritos bag and the student’s hand position as a potential firearm. School staff initially reviewed and canceled the alert, but a principal — unaware the alert had been cleared — notified the school resource officer, who called police. Omnilert, the vendor that runs the camera-based detection system, expressed regret but said the “process functioned as intended,” highlighting a breakdown in human–system communication rather than only model failure. The incident underscores two technical and operational risks of deploying real-time object-detection systems in sensitive settings: false positives from visual confusion (e.g., chip bags or hand posture resembling a weapon) and fragile alert-handling workflows that can amplify harm. From a model perspective this reflects precision/recall tradeoffs, class imbalance, and limitations in training data or feature representations that misclassify benign objects under occlusion or uncommon poses. Practically, it signals the need for better calibration, explainability (e.g., heatmaps or confidence scores), on-device human review protocols, robust cancelation/notification flows, and ongoing auditing to measure real-world false-positive rates before wide deployment.