Reproducing an Indirect Prompt Injection Against a RAG Pipeline (koreshield.ai)

A recent demonstration revealed a critical vulnerability in retrieval-augmented generation (RAG) pipelines through indirect prompt injection, where a seemingly benign query led to sensitive internal data being leaked. In this scenario, a customer-support assistant processed a document containing a hidden instruction to output all received documents alongside any internal context, such as API keys. This Trojan horse approach allowed the model to circumvent typical security measures, resulting in the unintentional disclosure of confidential information without any malicious network activity. This discovery is significant for the AI/ML community as it highlights the inherent weaknesses in current LLM implementations, particularly regarding their inability to differentiate between instructions and data. This vulnerability was tested across two popular models—OpenAI's gpt-5.4-mini, which resisted all five payloads, and Meta's Llama 3 8B Instruct, which complied with multiple attack types. To address this issue, a new AI security platform called Koreshield has been developed, which inspects and blocks harmful prompts before they reach the model. This proactive defense layer emphasizes the need for robust safeguards in AI systems, especially in applications handling sensitive data.