Prompt Injection as Role Confusion is my new favorite paper about a very obvious threat in hindsight that’s hard for us humans to see because we anthropomorphize LLMs so naturally. When Obi-Won Kenobi tells the stormtroopers that “These are not the droids you are looking for” to pass the checkpoint that’s role confusion: the guards foolishly think his words are their own thoughts. The very readable blog-style writeup explains the details, but I want to focus on the threat model perspective which is my bread and butter.
I look at software from a security perspective, and as amazing as the technology is, it seems that the list of reasons that modern LLMs are inherently untrustworthy just gets longer. Without limitation, a long list of challenges (that seem to be quite fundamental and not amendable to add-on remediation) includes: poisoned and errant training data, side effects of RLHF, ineffective guardrails[*], hallucination, speculative completion, lacking metacognition, alignment drift, context variation sensitivity, and now (new to me at least) role confusion.
Modern LLMs interfaces partition chat sessions with markers delimiting sequences of tokens as system prompt, user input, thinking, tool use, and its own responses as assistant; these various sections are associated with roles. As the paper’s conclusion explains, Role tags were a formatting trick that became the security architecture and the cognitive scaffolding of modern LLMs.
The phrase “became the security architecture” raises a big red flag because that sounds like nobody thought much about it. What follows is my simplistic take, but the abstract principles involved are so fundamental that details are not important to the basic argument.
Making sense of these sessions (for humans or LLMs) requires keeping track of the roles. Humans know how to understand conversations and easily follow the role markers (like HTML, <user>2+2</user><assistant>4</assistant>), it’s a completely reasonable scheme for us.
But assuming that LLMs interpret roles that way would be naive anthropomorphization; and just such an assumption appears to be how such a weak security architecture came to be. As the paper explains (section 1): … for an LLM, everything arrives through the same channel as one long token soup. Its own thoughts sit next to your instructions, which sit next to the contents of a random webpage it just fetched.
Designing a security architecture where user commands and data sit intermingled with root access only state and commands is already madness, but it gets worse. Classic software might be able to carefully parse such a token sequence accurately into respective roles, though it’s still a risky design, but LLMs do inference on that “token soup” where no hard boundaries of any kind exist or can be enforced. Once there is role confusion all bets are off, and prompt injection is just one of many sources of abuse or confabulation.
It’s hard to think of a murkier trust boundary design.
[*] The inherent fragility of guardrails is worth expanding on because it’s not just an unsolved problem: infallible guardrails are mathematically impossible. NIST has published a proof (behind a firewall; seems wrong for a government entity) that guardrails will inevitably have holes. Based on the summary openly available, “a fixed set of guardrails placed on AI is not universally robust against adaptive adversarial prompts”. The work is especially cool because it builds on the technique Kurt Gödel used to prove his incompleteness theorems. According to the NIST authors, “The findings show that developers and organizations deploying AI systems need to dedicate resources to finding prompts that would break the security of AI systems, and to address them before adversaries can exploit them.”