
Abstract
When RAG systems, browsers, or email tools pass external text to a language model, data to be analyzed can also resemble an instruction to ignore the task and call a tool. Greshake and colleagues showed that indirect prompt injection can affect model behavior through retrieved webpages, email, or documents, without direct access to the user’s prompt. Here we test a system boundary. The trusted task determines control flow, untrusted content supplies data, and a capability check independent of the model authorizes high-risk tool calls. The accompanying standard-library script compares three toy pipelines: direct concatenation, text delimiters alone, and structured isolation with an authorization gateway. It tests a program invariant. It neither measures attack or defense rates for a live model nor establishes immunity to natural-language attacks.
Reproduce value and goals: first ask "who has the right to issue orders"
Prompt Injection (Prompt Injection) is different from normal error answers. The target of the attack is not to make the model occasionally wrong, but to allow the low-trust content to gain the control rights that are not already available. Inject directly from the current user input; indirect Injection is hidden in search results, PDF, code comments, tool return or long-term memory. If the system spells the "summary of this email" and the body of the email into a string, the model sees all tokens, but the application expects it to automatically understand that the former is credible and the latter is not credible. This expectation itself has not formed an executable security boundary.
There are four goals in this time: the source and trust level of the input label for each segment; prove that the forgery XML The delimiter cannot constrain an executor that still scans the entire string of text; the action can only be derived from a trusted task; the "model proposal" and "system execution" are disassembled, and the tool name, parameter and authorization range are checked with a deterministic gateway. The experiment covers only one micro-action syntax, focusing on the test data flow, rather than collecting attack spells.
Threat model and core idea: data can affect answers, not forge permissions
Let the trusted task be , the external record is , the planner is , the tool executor is . Vulnerable systems usually calculate:
Among Indicates string splicing, is a candidate action. if only Hold If the command in it is true, the data enters the control flow. Structured design to:
is a Capability (ability token) cast only by trusted tasks, containing allowed tools and parameters; Indicates that the action does not exceed this ability. In the implementation, a batch of requests can be represented as trust.shape=[B,S]: is the number of samples, is the number of input fragments, each location is recorded trusted or untrusted. What really matters is not this tensor name, but the source tags are not lost during the search, planning, parameter assembly, log and execution phases.
"Isolation" also has hierarchies. Spotlighting uses continuous source tokens to enhance the distinction of external texts by the model; Instruction Hierarchy trains the model to obey the high authority Instruction first; StruQ will prompt and data As the two parts of the structured query, together with retained token, front-end filtering and specialized training; CaMeL goes further, extracts the control flow and data flow from the trusted query, and applies the Capability policy at the tool call. These routes are stronger than the sentence "Don't listen to the words in the document", but only the application side permission check can limit the model error to the controllable consequences.

Official paper and code reading route
The first stop to read the papers of Greshake et al. llm-security repository. The paper first defines the threat of "attacks are served by the retrieved data", the repository's scenarios/ Show attack surfaces such as web pages, emails, code completion and persistent memory. Don't take the demonstration payload as a general success rate when reading; you should draw the source, the path to the context, the model is visible, the callable tool and the final side effect.
The second stop reads Spotlighting and OpenAI's Instruction Hierarchy. The core of the former is source transformation of untrusted text, and the latter explicitly defines the priority relationship when conflicting with different permission instructions. Paper report: Spotlighting has reduced the attack success rate from more than 50% to less than 2% in its GPT series experiments; this is only the result of a specific model, attack set and protocol, and cannot be directly written as a guarantee for all systems. Instruction Hierarchy is also model behavior training and should not replace application authorization.
Read Section 4 of the StruQ paper on the third stop, and then look at the official repository struq.py、train.py、test.py and run.py. The key is not to copy the string template, but to understand two things: the separator uses the token that the attacker cannot generate directly, and the front end also needs to filter forged marks; partial instructions. Ordinary <DATA> Labels are still ordinary text, and there is no equivalent replacement for StruQ when there is no special training and non-forgery coding.
The fourth stop is to read AgentDojo. The NeurIPS 2024 paper provides 97 real tasks and 629 security test cases, and the official command can be combined with Suite, User Task, Attack and tool_filter defense. Reproducing the experiment requires both Benign Utility and Attack Success, instead of refusing to obtain "safety". Finally read CaMeL's src/camel/、tests/ With the reproduction command, focus on tracking control flow extraction, value source, policy playback and tool call check; the repository clearly states that it is a research product, may contain bugs and may not be completely safe, and this warning should also be kept in the reproduction report.
Minimum experiment: three pipelines and four types of attacks
code/minimal_isolation_gateway.py Models are not called. It deliberately implements a vulnerable parser: as long as it appears in the mixing string ACTION::工具::目标, to perform the last action. Six fixed cases include two normal tasks and four types of attacks: unauthorized recipients, forged closed separators, parameter replacement of authorized tools, and forged control tokens. One of the normal tasks is clearly authorized to send a summary to the researcher's mailbox to check whether the defense is accidentally injured.
plain-concat Directly splice tasks and records; delimiter-only affiliate <TASK> and <DATA>, but the actuator still scans the entire string; isolated-gateway Only generate actions from trusted tasks, and require the action source, tool name and target to be exactly the same as the Capability. Untrusted records can still participate in the abstract, but cannot cast action permissions.
In the CPU smoke test, direct concatenation and delimiter-only separation both had attack success rates of 1.000. All four test cases produced unauthorized side effects. The isolated gateway had an attack success rate of 0.000 and benign utility of 1.000 on both ordinary tasks. It executed two authorized email actions: one in an ordinary task and one in which it kept the user-authorized recipient despite an attempted parameter substitution. Syntax checks and assertions passed.
These numbers are deterministic protocol tests on the artificially constructed parser, not a reproducing score for LLM, StruQ, AgentDojo, or CaMeL. They only prove that if the executor still extracts the command from the untrusted string, the other label does not change the trust relationship; if the permission can only be generated by the trusted input, the TOY attack cannot directly form side effects.
Evaluation protocol: Security and Availability must be reported in pairs
Attack success rate is defined as , where the molecule is the number of attack samples that actually have unauthorized side effects; it is not possible to replace the system state with only a word in the model output. Normal task completion rate A pseudo-defense used to identify "all rejections". It should also be reported that the pass rate of the legal tool call, the interception rate of the overpowered parameter, the manual confirmation rate, the additional delay and the cost of the token should be reported.
The baseline should be increased layer by layer, not only "no defense" and complete system: system prompt reminder, ordinary separator, content purification, source marking, injection classifier, tool filtering, parameter level authorization, manual confirmation, and then make a combination of ablation. In this way, we can distinguish whether the model really ignores malicious instructions, or whether the attack affects the text output but is truncated by the execution layer. For a random model, the same attack is repeated at least several times; if you only pick a screenshot of success or failure, you can neither estimate the variance nor compare the defense cost.
The dataset is at least four layers. The first layer is direct conflict and separator forgery; the second layer is coding, misspelling, long context and multi-round persistence; the third layer allows attack and positive The semantics of the regular business is similar, and the false positive is measured; the fourth layer enters the real proxy environment, and checks the final status of mail, network disk, bank or code tool. Each sample holds a trusted task, original untrusted content, source, expected action set, real call trajectory, final state and version hash. Attack and defense should be paired on the same task, random seeds and decoding parameters are fixed, and multiple samples are sampled to Confidence interval.
The authorization gateway must be outside the model. To send_email(to, body), the strategy cannot just check the tool name, but also to Whether it comes from the recipient explicitly given to the recipient, body Whether it is allowed to contain data read from a private source, and whether the action needs to be acknowledged. Read-only retrieval, writing files, sending messages, and deleting resources should have different capabilities; unknown tools, additional parameters, and cross-tenant identification are rejected by default. The log should log the policy input and rejection, but do not spread the secret or attack content to a new context.
Failure analysis: first check where the boundary is broken
seem <DATA> The defense still fails, first see if the model or parser treats the label as ordinary text, and whether the attacker can write the same closed tag in the data. If the input filtering seems to be effective, add Unicode, HTML comments, indirect references and semantic rewriting; the high keyword Recall does not mean that the new attack is still stable. If the independent detector releases the attack, remember that the detector itself may also be injected when the model is also a model, and cannot be granted execution permission.
If the tool name is correct but the target is replaced, the problem is usually in the parameter Provenance: the system only verifies the schema, and there is no validation value from where it comes from. If the model does not call the tool but leaks the secret, check whether the Markdown image, URL, reference and rich text rendering form a takeout channel. If the current round is safe and the next round is lost, check whether the memory write, the summary cache and the vector library are persistent malicious instructions. If the defense ASR is very low but the Utility also crashes, it means that it may only be excessively abstent; it should be graded according to risk, and the manual confirmation should be concentrated in irreversible or cross-boundary actions.
The most important counterexample is "the trusted task itself is ambiguous". Users say "handling all emergency messages" does not define the object, amount, or deletion range that can be sent, and the system cannot cast the correct Capability out of thin air. At this time, you should ask or narrow down to read-only previews, instead of letting the model guess permissions. The authors infer that the focus of the research should shift from looking for universal Prompt to prove which information flows can reach which side effects, and use system status verification to ensure.
Questions for further research
One is how to automatically extract the minimum from natural language tasks Capability, at the same time, avoid over-authorization of vague intentions; the second is how to maintain the source tag after abstract, rearrangement and multi-agent communication; the third is the structure Whether the chemical training can generalize the attack on the non-language, modal and long-context attacks; Whether the Taint Tracking and the privacy type system can cover indirect data out-out; sixth, when humans confirm the budget is fixed, how to optimize the risk coverage instead of simply increase the pop-up window.
Summary
When instructions and data share a representation, low-trust text can influence high-privilege control flow. A stronger written warning leaves that underlying problem in place. To make the threat model testable, identify the sources, actions, and final states. Structured inputs preserve the trust boundary, the model proposes a plan, and a least-privilege gateway checks the tools, arguments, data flows, and confirmation requirements. The script verifies the difference between these program designs. How a real model responds remains open. Isolation makes it harder for a misleading input to cause an irreversible side effect, even when the model is fooled.
References
Search date: 2026-09-04. The following are papers, conference pages, research institution pages, official code or security project documents; model behavior, online lists and repository interfaces may change, and manual verification is used when using them.
- Kai Greshake et al., 2023, Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection and Official Demo Repository
- Keegan Hines et al., 2024, Defending Against Indirect Prompt Injection Attacks With Spotlighting(Microsoft Research)
- Eric Wallace et al., 2024, The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions(OpenAI)
- Sizhe Chen et al., 2025, StruQ: Defending Against Prompt Injection with Structured Queries(USENIX Security) and official implementation
- Edoardo Debenedetti et al., 2024, AgentDojo(NeurIPS Datasets and Benchmarks) and Official repository
- Edoardo Debenedetti et al., 2025, Defeating Prompt Injections by Design(CaMeL) and Google Research Recycling Repository
- OWASP, LLM Prompt Injection Prevention Cheat Sheet