Invisible Prompt Injection Through Markdown and HTML-Comments (github.com)

A new study reveals a critical vulnerability in Markdown documentation that exposes AI systems to "invisible prompt injection," where malicious instructions can be embedded in HTML comments and Markdown reference links. These hidden elements, unseen by developers, can influence the AI when it processes raw Markdown, leading to the integration of unsafe code during deployment. The Finnish cybersecurity consultancy Bountyy Oy highlights this concern and introduces the Documentation Rendering Parity Test (DRPT) as a benchmark to evaluate AI models’ handling of such hidden content, showing significant risks across existing AI systems. This research is significant for the AI/ML community, as it shifts the focus from model alignment to preprocessing failures within documentation. The proposed Safe Markdown for AI Consumption (SMAC) standard outlines methods to sanitize Markdown before it reaches AI models, effectively closing the vulnerability by stripping out potentially harmful content. By adopting these techniques, platforms and IDE copilot teams can safeguard against attacks leveraging manipulated documentation, underscoring the importance of thorough inspection of all inputs to AI systems.