HuggingFace Chat Exfiltrates Data (www.promptarmor.com)

HuggingFace's chat application, HuggingChat, has been discovered to have a serious security vulnerability that allows for zero-click data exfiltration through indirect prompt injection. This flaw enables attackers to embed malicious instructions within various data sources, such as documents or web pages, which can manipulate the AI model to generate unsafe Markdown images. When rendered in the chat, these images can exfiltrate sensitive user information, like financial data from uploaded documents, without any user interaction. The discovery was responsibly disclosed to HuggingFace for mitigation, but as no response was received, the issue has been made public to alert users. This vulnerability underscores significant risks associated with AI models that interact with untrusted external sources. It highlights the potential for serious data breaches resulting from model outputs that appear benign but harbor malicious prompts. Technical recommendations to mitigate these risks include prohibiting the rendering of Markdown images from external sites unless explicitly confirmed by the user and implementing a robust Content Security Policy to block unauthorized network requests. Addressing these issues is critical for ensuring the security and privacy of users engaging with AI systems in the future.