Demystifying Security Risks of AI-Powered Applications on Pre-Trained Model Hubs (arxiv.org)

A systematic security analysis of AI-powered applications (AI-Apps) hosted on platforms like Hugging Face has uncovered significant vulnerabilities that pose risks to users and developers. The research identifies five threat categories and ten attack vectors, revealing critical security flaws such as broken access control, insecure resource reuse, and insufficient input validation. Notably, the study highlights three new architectural vulnerabilities specific to these platforms, which exacerbate traditional security issues. This analysis is significant for the AI/ML community as it sheds light on the hidden security risks of leveraging pre-trained models in a democratized environment. By examining over 970,000 public AI-Apps, the researchers identified alarming instances of credential leaks, input injection vulnerabilities, and embedded backdoors. The findings emphasize the urgent need for enhanced security measures and proper configurations within these platforms to protect sensitive data and ensure the integrity of AI-Apps. The responsible disclosure of the findings to affected parties indicates a proactive approach towards improving security in the rapidly evolving AI landscape.