Voice AI Systems Are Vulnerable to Hidden Audio Attacks (spectrum.ieee.org)

New research presented at the IEEE Symposium on Security and Privacy reveals that AI-powered voice and audio systems are vulnerable to hidden audio attacks, where imperceptible sound modifications can manipulate these systems into executing unauthorized commands. Conducted by researchers from Zhejiang University, the study highlighted a technique known as AudioHijack, which boasts an alarming success rate of 79 to 96 percent. This method exploits a security flaw in large audio-language models (LALMs) by embedding malicious instructions within audio clips, allowing attackers to hijack models without needing control over the original user commands. Their tests targeted 13 leading open models, including commercial services from Microsoft and Mistral, revealing capabilities to perform actions such as illicit web searches, file downloads, and email transmissions. The significance of this research lies in its potential implications for both security and the design of AI systems. It underscores a critical vulnerability that extends beyond traditional adversarial audio attacks, challenging the underlying architecture of generative AI models. The study reveals that, unlike older models, generative models struggle to detect manipulated audio due to their complex processing of audio into numerical tokens. As the AI/ML community grapples with these findings, it raises urgent questions about the resilience of generative systems and the necessity for robust defenses against such insidious attacks, with limitations of current protective measures highlighted by the researchers.