A Theory of Why Prompt Injection Works (role-confusion.github.io)

A recent study has proposed a theory to explain the phenomenon of prompt injection in large language models (LLMs), highlighting a fundamental flaw in how these models perceive different roles within a conversation. This research demonstrates that LLMs struggle to accurately differentiate between their own thoughts and external inputs because they process all information as a continuous string of text, lacking the distinct sensory channels humans use. To impose structure, LLMs rely on role tags (like system, user, think, and tool) that are intended to guide their understanding of context and authority. However, the overloading of these role tags leads to vulnerabilities, allowing attackers to insert commands disguised as benign text, effectively hijacking the model's response mechanisms. The significance of this work lies in its insights into the limitations of current model defenses against prompt injections. The research identifies two primary strategies for LLMs to resist such attacks—attack memorization and accurate role perception—where the latter proves more robust. By developing a method termed "role probes," the researchers can quantitatively measure how well LLMs recognize different roles internally. These findings pave the way for a deeper understanding of role dynamics in LLM cognition and provide a foundation for future research, suggesting the need for a more nuanced approach to enhancing model security against adversarial inputs.