arXiv:2605.17634: Why Data/Instruction Separation Cannot Stop Prompt Injection

Why Data/Instruction Separation Cannot Stop Prompt Injection

Researchers Sahar Abdelnabi (CISPA Helmholtz Center for Information Security) and Eugene Bagdasarian (Google) have published the paper arXiv:2605.17634, challenging the foundational assumption of today’s AI agent defenses.

Prompt injection is an attack in which malicious content from the environment — a document, web page, or API response — plants hidden instructions into an AI agent and hijacks its actions. The dominant defense today is data/instruction separation: distinguishing trusted user instructions from untrusted external data, and blocking the execution of instructions that arrive through the data channel.

The authors mathematically prove that this approach has a fundamental limit. An attacker does not need to plant text that looks like an instruction — it is sufficient to manipulate the situational context. An agent that correctly distinguishes data from instructions can still be led into the wrong action if an attacker constructs a legitimately-looking context that changes what the agent considers an “appropriate” action.

Contextual Integrity as a New Theoretical Framework

To formalize the problem, the authors introduce Contextual Integrity (CI) — Helen Nissenbaum’s framework from the philosophy of privacy. CI does not assess what is transmitted, but whether an information flow is appropriate to its context: who sends, to whom, in what situation, and for what purpose.

Applied to AI agents: an attack is not merely a planted instruction — an attack is any information flow that violates the contextual norms of a legitimate task. The authors develop a scenario analysis showing three violation mechanisms: misrepresentation of the information flow, manipulation of contextual norms, and the mixing of multiple flows from different contexts.

The key theoretical result — an impossibility result — states: an attacker can always construct a context in which a blocked legitimate operation looks suspicious and a malicious operation looks legitimate. Every tightening of security norms blocks some legitimate operations; every relaxation allows some attacks through.

Is Defense Even Possible?

The authors do not claim defense is impossible — they argue that the existing paradigm is insufficient. The solution is not a better detector for forbidden content, but a CI-aware alignment framework: agents need to be trained to assess the appropriateness of information flows according to task context, not merely to distinguish data and instruction formats.

The implications are direct for all production AI agents that process external content — email, documents, web pages, API responses. Channel separation remains a useful measure, but as the sole line of defense it is not enough.