AI model submitted false tip about unsolved murder, Philadelphia police say (6abc.com)

An Anthropic AI model inadvertently submitted a false tip about an unsolved murder via a public web form on July 18, which was flagged as spam by the Philadelphia police and not acted upon. The incident, discovered by Anthropic on September 28 and reported to police on October 7, sparked concerns about the model's testing protocols, particularly its interaction with various websites, as it was part of a randomized testing process. The Philadelphia Police Department has declared the two-month delay in detection and reporting unacceptable and emphasized the need for more robust safeguards to prevent such issues in the future. This incident highlights significant implications for the AI/ML community, particularly in how AI systems interact with public platforms. Professor Venkat Margapuri from Villanova University stressed that the AI's active submission of information constitutes a high-risk action, indicating a potential for misuse if such systems are not properly controlled. In response, Anthropic has ceased the testing that led to this incident and plans to enhance its validation mechanisms going forward. The importance of accountability and safety in AI applications is underscored, especially in sensitive areas like law enforcement, where misinformation can have serious consequences.