Cover illustration

Date:2026-06-26
Who this is for: Graduate student, LLM application researcher, security engineer, Agent/RAG system developer
search date:2026-06-26

Abstract

Prompt Injection is like a "Prompt project" problem: the user writes a sentence "Ignore the above" in the input, and the model may deviate from the sender's intention. By 2025-2026, this issue has evolved into the core topic of Agent Security: LLM is no longer just generating text, but in browsers, mail, code repository, MCP Scheduling permissions between tools, databases, and long-term memory. What is really dangerous is not the "wrong thing" of the model, but the untrusted page, mail or tool description affects the action parameters with permissions, such as the recipient, transfer account, command line, file path, API Endpoint.

The main line of research in the past year is very clear: attacks shift from direct injection to indirect injection, tool poisoning, memory poisoning and multistep state pollution. The collection moves to real task environments, browser agents and stakeholder perspectives; defense from "filtering malicious text" to instruction hierarchy, structured channel, ability sandbox, source tracking, minimum permissions, and runtime authorization. The most important inspiration for the project is: don't use LLM as a security boundary, the security boundary should be at the tool, data flow, permissions, and auditable execution layer.

Contents

  • Research background: Why Agent amplifies injection risk
  • roadmap for the past year
  • Key papers by research question
  • Comparing the methods
  • Technical trends
  • Implications for deployment
  • Limitations and open questions
  • Questions to watch
  • References

Research background: From Prompt to permission confusion

The root cause of Prompt Injection is that "instructions" and "data" share the same natural language context. In traditional Web security, SQL Injection is dangerous because data is interpreted as code; in LLM applications, external text is interpreted as a higher priority task instruction. Simon Willison named this type of problem Prompt Injection in 2022; Greshake et al. Indirect Prompt Injection in 2023 The paper further points out that the attacker does not have to directly enter a malicious Prompt to the model, as long as the instruction is hidden in a Web page, email, document or search result, the application may be hijacked when reading external data.

Agents raise the stakes. A chatbot’s mistake may be confined to its text, while an agent can read email, send requests, write files, or call a payment or deployment API. The security question becomes who can authorize which arguments in which tool calls. OWASP’s 2025 list ranks prompt injection first among LLM application risks and also addresses excessive agency, sensitive information disclosure, supply chain risks, and risks associated with tool use. Official MCP security guidance highlights user consent, least privilege, the confused deputy problem, and token passthrough risks. The NCSC warned earlier that prompt injection may be intrinsic to LLM technology and cannot be eliminated through stronger prompts alone.

roadmap for the past year

research roadmap

The focus of 2024 is to define the problem and create the environment: Instruction Hierarchy divides the system, developers, users, tools/third-party content into different permission levels; StruQ uses structured query to train the model to only obey Prompt Regional instructions; AgentDojo puts mail, bank, travel and other tool tasks into a scalable environment, and evaluates the attack and defense of the agent under untrusted data.

In 2025, research has clearly shifted to "real Agent scenarios". WASP focuses on the Web Agent, which limits the ability of the attacker to the conditions that are closer to the content manipulation of the Web page; CaMeL parses trusted user requests into control flow and data flow, and then uses Capability to restrict untrusted data to affect the program process; MCP Safety Audit uses the MCP ecosystem as a new attack surface, discussing the risk of malicious tools, tool descriptions of poisoning, credential stealing and remote execution.

A more fine-grained systematic study appeared in the first half of 2026: AgentPI and SoK/benchmark emphasize the context-dependent task, which is the real agent The environment context must be read to complete the task, and simply isolating external content will damage the utility; preprints such as PACT, AuthGraph, SafeAgent, and SecureClaw will focus on the parameter level. Provenance, authorization diagram, runtime controller, and read and write double borders. They also need more independent reproduction, but the direction is worth paying attention to: security is not just the input classifier, but executing permission constraints on the trajectory.

Key papers by research question

1. Attack: from direct injection to state pollution

Direct injection attacks usually come from user input, and the goal is to override the system or developer instructions; Indirect injection attacks come from external data, and the threat is closer to the real deployment. The classic conclusion of Indirect Prompt Injection is that as long as the application puts a web page, mail or retrieved documents into the same context, the attacker may let the model perform the action of "the user is not authorized but the agent has the ability to do it".

The attack surface has continued to expand in the past year. Web Agents need to parse page visual elements, hidden text, forms and navigation states; MCP Agent needs to trust the exposed Tools of third-party servers Descriptions; Assistants with long-term memory may write malicious rules into future sessions. ASB includes system prompt, user prompt, tool call and memory retrieval in the attack stage, indicating that the agent security must look at the complete life cycle, and not only do one filter at the entrance.

2. Benchmark: from static collection to task environment

Prompt Injection Early Benchmark is often like a classification question: give a text to determine whether the model is induced. AgentDojo's contribution is to combine tasks, tools, and attacks into dynamic environments, and distinguish Task Utility from Security. The 97 real tasks and 629 safety test cases are listed in the public abstract of the paper, which enables researchers to compare the tension between "whether normal tasks can be completed" and "can resist attacks".

WASP further focuses on browser agents. Its summary reports an interesting phenomenon: some agents will start executing malicious instructions at 16%-86%, but the proportion of truly end-to-end attack targets is only 0%-17%. This is not a victory of security, but a reminder that the current Agent's lack of ability will "absolutely reduce the attack completion rate"; once the basic ability is improved, the same security design may quickly fail.

The 2026 StakeBench/AgentPI work extends the focus from attack success to victim and context dependence. An injection can damage users, sellers, platforms, or third parties at the same time; and defense can prevent the agent from completing authorization tasks if the environment context is roughly blocked. In the future, BenchMark should also report task success, attack success, error rate, delay, authority over-border type and auditability at the same time.

3. Defense: Three routes are taking shape

The first route is the model side Instruction priority. Instruction Hierarchy makes the model recognize the high and low priority conflicts through training; StruQ separates the "Instruction area" and the "data area" with a structured channel. They are suitable as basic capabilities, but cannot bear the system security boundary alone. In 2025, the preprint of the white-box attack on the FINE-TUNING defense shows that structured or aligned defenses may still be broken through under strong attackers.

The second route is the architecture side isolation. The core idea of CaMeL is that the trusted request determines the control flow, and the untrusted data can only be used as data, and should not change the program process; Capability is used to restrict data outflow. This kind of work is most valuable to engineers, because it rewrites the problem from "making the model more obedient" to "making the model have no permission to make critical mistakes".

A third research direction studies provenance and authorization at runtime. The 2026 preprints PACT, AuthGraph, and SecureClaw focus on where tool arguments come from. For example, the body of an email may inform a summary, but it should have no authority to choose a transfer account or an outgoing email address. This line of work draws on information flow control, taint tracking, and policy enforcement and could become part of agent platform infrastructure.

Comparing the methods

Category Representative work Main object Advantage Risk and Points to Verify
Instruction level training Instruction Hierarchy model behavior Let the model understand permission conflicts Cannot replace tool authorization
structured channel StruQ Prompt/Data Separation Clear thinking, easy to explain The robustness of the white box attack still needs to be verified
Dynamic Evaluation Environment AgentDojo, ASB Agent tool tasks Ability to measure Utility and Security at the same time Ecological mobility of different tasks is limited
Web Agent Reviews WASP, WAInjectBench, StakeBench Browser agent Closer to real web attacks Page complexity and model version change fast
ability sandbox CaMeL Control Flow, Data Flow, Capability Move the security boundary out of the model Engineering costs and task coverage are challenges
Source/Authorization Map PACT, AuthGraph, SecureClaw Tool parameters and read and write boundaries Close to the permission system, can be audited Mostly 2026 preprints, to be reproduced independently
Agreement/Ecological Audit MCP Safety Audit, MCP-38 MCP Server and Tool Covering the Emerging Tool Eco The actual MCP deployment is very different

Technical trends

First, the Prompt Injection is changing from "input security" to "execution security". It is not enough to judge an external text as malicious, because the content of the same web page is data in the summary task, and it may become a dangerous parameter in the transfer task. Security policies bind tasks, tools, parameters, and sources.

Second, Agent Security is absorbing traditional Security paradigms: Least Privilege, Confused Deputy, Capability, Taint Tracking, Policy-As-Code, audit logs and human confirmation. The LLM is responsible for semantic understanding, but the permission system must be performed by deterministic components.

Third, the MCP and tool ecology will become high-value research objects. MCP reduces tool access costs, and also makes tool descriptions, server identity, authorization Scope, token delivery and multi-server combinations a new boundary. Tool descriptions should not be asserted by default as trusted facts, and supply chain governance is required for installation, update, authorization, and invocation.

Fourth, the evaluation will shift from "whether the attack is successful" to "who is damaged, how it is damaged, and whether it can be held accountable". The real platform cares not only about ASR, but also Benign Task Completion, misintercepting costs, user confirmation burden, leak path, recovery ability and post-hoc evidence collection.

Implications for deployment

  1. Label all external content as untrusted. Web pages, mail, PDF, RAG chunk, tool return, MCP tool description, and long-term memory should not be automatically upgraded to instructions.
  2. Do parameter-level authorization for tool calls. Allowing agents to summarize messages does not mean that the body of the mail is allowed to decide the recipient, attachment, transfer destination, or shell command.
  3. Use minimum permissions and staging submissions. High-risk actions use PREVIEW -> USER_APPROVE -> COMMIT; the COMMIT phase is a normalized request by a trusted executor.
  4. Record Provenance. Each critical parameter should be traced back to user input, system configuration, external data, or model generation; audit logs should cover read, inference, tool calls, and final output.
  5. Supply chain governance for MCP/Server. Fixed server version, validation publisher, restricted scope, prohibits silent expansion rights, and avoids the use of tool descriptions as natural and trustworthy.
  6. Don't take "System Prompt is more strictly written" as the main line of defense. Prompt can reduce misoperation, but cannot provide strong security guarantees.
  7. Evaluation should include true failure modes. At least cover direct injection, indirect injection, tool poisoning, memory contamination, data breach, overwritten write operation, and Benign Utility.

Limitations and open questions

There are still three obvious gaps in the current research. First, many papers are evaluated on different models, tasks and tool ecology, and the results are difficult to compare horizontally; the same Defense is valid on the mail agent, which does not mean that it is valid on the code agent or browser agent. Second, in 2026, many system architecture papers are still pre-printed, and the reported 0% ASR or high Utility numbers need to be independently reproduced, open source and stronger Adaptive Attack. Third, the contradiction between security and availability has not been resolved: the complete isolation of external data will make the agent unable to complete the context dependence task; over-trust context will bring permission confusion.

Therefore, this article does not regard any single solution as the final answer. A more realistic deployment form may be a combination: the model-side instruction hierarchy is the basis, the tool layer has the minimum authority as a hard boundary, and the Provenance/Authorization time is used as the audit and interception layer, and the high-risk action is handed over to humans for confirmation.

Questions to watch

  • Will there be a general-purpose information flow control framework for agent-oriented, like the CSP/permission model in Web security to become the default capability of the platform?
  • Will the MCP ecosystem form verifiable tool identity, permission statement, signature update and security auditing specifications?
  • How can long-term memory be revoked, interpretable, and isolated?
  • How can the visual injection, hidden text and cross-page state pollution of Web Agent be systematically evaluated?
  • Can the evaluation indicators expand from ASR to Stakeholder Harm, mistakes in cost and resilience?

Summary

Prompt Injection's research focus has shifted from "whether the model will listen to the attacker's words" to "whether the authority of the agent is indirectly controlled by untrusted data". The most important trend in the past year is not a higher Detector score, but the security boundary is moving out: from Prompt to structured context, from model to tool runtime, from text classification to parameter source and authorization relationship. For the engineering team, the safest route is to admit that LLM is not a security boundary, and regard the Agent as a software system with permission, state, and supply chain to design.

References

Search date: 2026-06-26.

  1. Simon Willison, “Prompt injection attacks against GPT-3”, 2022. https://simonwillison.net/2022/Sep/12/prompt-injection/
  2. Kai Greshake et al., “Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection”, arXiv, 2023. https://arxiv.org/abs/2302.12173
  3. Eric Wallace et al., “The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions”, arXiv, 2024. https://arxiv.org/abs/2404.13208
  4. Sizhe Chen et al., “StruQ: Defending Against Prompt Injection with Structured Queries”, arXiv / USENIX Security 2025. https://arxiv.org/abs/2402.06363
  5. Edoardo Debenedetti et al., “AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents”, arXiv, 2024. https://arxiv.org/abs/2406.13352
  6. Hanrong Zhang et al., “Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents”, arXiv, 2024. https://arxiv.org/abs/2410.02644
  7. Ivan Evtimov et al., “WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks”, arXiv, 2025. https://arxiv.org/abs/2504.18575
  8. Edoardo Debenedetti et al., “Defeating Prompt Injections by Design”, arXiv, 2025. https://arxiv.org/abs/2503.18813
  9. Brandon Radosevich and John Halloran, “MCP Safety Audit: LLMs with the Model Context Protocol Allow Major Security Exploits”, arXiv, 2025. https://arxiv.org/abs/2504.03767
  10. Yinuo Liu et al., “WAInjectBench: Benchmarking Prompt Injection Detections for Web Agents”, arXiv, 2025. https://arxiv.org/abs/2510.01354
  11. Peiran Wang et al., “The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysis”, arXiv, 2026. https://arxiv.org/abs/2602.10453
  12. Zihao Wang et al., “Who Pays the Price? Stakeholder-Centric Prompt Injection Benchmarking for Real-world Web Agents”, arXiv, 2026. https://arxiv.org/abs/2606.13385
  13. Linfeng Fan et al., “The Granularity Mismatch in Agent Security: Argument-Level Provenance Solves Enforcement and Isolates the LLM Reasoning Bottleneck”, arXiv, 2026. https://arxiv.org/abs/2605.11039
  14. Peiran Wang et al., “Aligning Provenance with Authorization: A Dual-Graph Defense for LLM Agents”, arXiv, 2026. https://arxiv.org/abs/2605.26497
  15. Hailin Liu et al., “SafeAgent: A Runtime Protection Architecture for Agentic Systems”, arXiv, 2026. https://arxiv.org/abs/2604.17562
  16. Yuhan Ma and Stefan Schmid, “SecureClaw: Clawing Back Control of LLM Agents”, arXiv, 2026. https://arxiv.org/abs/2606.09549
  17. Yi Ting Shen et al., “MCP-38: A Comprehensive Threat Taxonomy for Model Context Protocol Systems”, arXiv, 2026. https://arxiv.org/abs/2603.18063
  18. OWASP, “LLM01: Prompt Injection”, OWASP Top 10 for LLM Applications 2025. https://genai.owasp.org/llmrisk/llm01-prompt-injection/
  19. OWASP Cheat Sheet Series, “LLM Prompt Injection Prevention Cheat Sheet”. https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html
  20. Model Context Protocol, “Security Best Practices”, specification version 2025-06-18. https://modelcontextprotocol.io/specification/2025-06-18/basic/security_best_practices/
  21. NIST, “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile”, NIST AI 600-1, July 2024. https://doi.org/10.6028/NIST.AI.600-1
  22. UK NCSC, “Exercise caution when building off LLMs”, 2023. https://www.ncsc.gov.uk/blog-post/exercise-caution-building-off-llms