Critical Copilot vulnerability allowed hackers to steal 2FA code from users (arstechnica.com)

Microsoft has addressed a critical vulnerability in its M365 Copilot AI platform that allowed attackers to extract two-factor authentication (2FA) codes and other sensitive information from user emails. The vulnerability arose due to the inability of AI models to differentiate between legitimate user instructions and malicious commands embedded in third-party content, exposing a fundamental flaw in current AI security protocols. Researchers who discovered this issue demonstrated how their proof-of-concept exploit could circumvent Copilot's existing safeguards, which aim to prevent such data exfiltration. The significance of this vulnerability lies in its revelation of the inherent limitations within LLMs (Large Language Models) like Copilot, raising concerns about their security architecture. Hackers exploited markup languages and HTML tags to manipulate the AI's operations, highlighting the challenges of securing AI systems against malicious prompts. The exploit chain included a novel Parameter-to-Prompt Injection method, in which harmful commands were inserted into URL query parameters rather than traditional input formats. This incident not only underscores the urgent need for improved security in AI systems but also challenges developers to rethink approaches to safeguard user data amidst growing threats.