Prompt injection: the attack classes and the defences that hold up

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

Prompt injection is what happens when text the developer did not write is obeyed as if it were the developer's instruction. The model has one token stream. System text, the user's message, a web page, and a tool result are all tokens in that stream. Models may be trained to respect instruction roles, but that behaviour is not a deterministic authorisation boundary. External controls are needed to contain an incorrect tool action.

An AI writing system produced this article for NetLeverage's editorial studio. Descriptions of attacks are the classes named in the cited papers, not procedures for carrying them out. Nothing here was executed. Success rates below belong to the papers' models and prompts, not to a current product.

Why it matters

A chatbot that only talks can be embarrassed by a bad answer. An application that reads mail, browses, or calls tools can be redirected into sending data or taking an action the user did not ask for. Greshake et al. put the change in one sentence: retrieval blurs data and instructions, so an adversary who never sees the chat can still reach the model by placing text where the application will fetch it.

Wallace et al. name the mechanism underneath the variants. System messages, user messages, and tool outputs arrive as text of different trust, and models that treat them as the same priority will follow the lower-priority text when the two conflict. Their analogy is an operating system in which every instruction runs as if it were kernel mode. SQL injection was reduced by refusing to treat user input as SQL. The corresponding refusal, for them, is an instruction hierarchy: system over user, user over third-party content.

How it works

Two places the untrusted text can enter

Anthropic's current guardrails page, opened 10 November 2026, splits the threat into two models, and the papers use the same cut.

Direct injection. The user of the application is the adversary. They type, or paste, text meant to override the application's instructions or a safety policy. Perez and Ribeiro studied this on GPT-3 (text-davinci-002) in 2022. They define prompt injection as inserting malicious text to misalign the model, and they split it into two goals.

• Goal hijacking. The original task is replaced by a new one, in their experiments the task of printing a chosen target phrase. A run counts as a success only if the output is exactly that phrase, ignoring punctuation and line breaks.
• Prompt leaking. The new goal is to print part or all of the application's original prompt. Success means the output contains the original instruction. They found leaking harder than hijacking on the same 35 public example prompts.

Those 35 prompts came from OpenAI's examples page, which they chose because developers copy them. Each configuration was repeated four times because the model is stochastic. Factors they measured, and which still matter as design facts rather than as a ranking of today's models: the wording of the attack changes the rate; delimiter characters change it, without a clear winner among delimiter length and repetition; temperature changes it, while in their tables top-p and the frequency and presence penalties do not; a stop sequence makes the attack harder; a prompt that still has developer text after the user-input slot is harder to hijack; more harmful target phrases succeed less often than a bland target. The most susceptible model in their comparison was text-davinci-002. Treat every percentage in that paper as a 2022 GPT-3 result.

Indirect injection. The user is trusted. The adversary writes the text that the application will retrieve: a web page, a message, a document, a repository file, a tool result. Greshake et al. call this out because the earlier assumption was that the attacker had to be the person at the keyboard. They group delivery into passive retrieval (including search-engine promotion of a poisoned page, and a sidebar that reads the page the user is already on), active delivery (for example, attacker-sent email read by an assistant), user-driven delivery (the person pastes a copied snippet that carries the extra text), and hidden delivery (a small first instruction that tells the model to fetch a larger payload, or text carried in another modality, or text that is encoded so a keyword filter does not see it). They demonstrated encoded delivery against Bing Chat in that paper. The lesson they draw is that a filter applied only to what the user typed does not see the retrieved channel.

They sort impact with a security taxonomy rather than a list of phrases: gathering information (including persuading a person, or an assistant that can read and send mail, to move data), fraud, malware, intrusion and persistence through tools and memory, manipulation of the answer the user trusts, and availability loss, including disruption of tool calls. Their fifth key message is the one that connects to agent loops: when the model decides which API to call and then reads the result, both the outbound call and the inbound result are points of manipulation.

Jailbreaking sits next to these and is not the same goal. Wallace et al. define a jailbreak as an attempt to escape safety behaviour trained into the model, often without contradicting a specific application instruction. Prompt injection, in their wording, targets the application built on the model: the business rule, the secret, the tool. A single user message can try both. Defences that only look for one class miss the other.

Why a delimiter loses

The model does not have a parser that treats a quoted span as data the way a SQL driver can bind a parameter. Perez and Ribeiro observed that delimiter characters can make an attack more effective, which shows they are not a trustworthy semantic boundary. Greshake et al. argue that input filters on retrieved text are brittle, and that encoding is one way past a filter that looks for phrases. They also describe the trap in using a second model as a filter. A weaker model, not trained to follow instructions, may miss an encoded payload. A stronger, instruction-following model can be injected by the same payload it was asked to detect. A supervisor that never reads the retrieved text will miss attacks whose harm depends on that text, including planted falsehoods. Their conclusion in that paper is that they cannot see a foolproof countermeasure, and that robustness against evasion still had to be measured.

That is why "tell the model to ignore instructions in the document" is a measure that may reduce success rates, not a boundary. It can reduce how often the model complies. It does not make compliance impossible.

What training can change, and what it costs

Wallace et al. train a hierarchy into GPT-3.5 Turbo with supervised fine-tuning and reinforcement learning from human feedback. The rule they want is conditional. A lower-priority instruction that agrees with the higher one should be followed. Their example is a car-sales bot told "use Spanish": that is aligned with the system role, and the model should comply. A lower-priority instruction that conflicts should be ignored, or refused when there is no way to continue.

They build the data in two ways. For aligned requests they decompose a compositional instruction ("write a 20-line poem in Spanish") into pieces, place the pieces at different levels, and train the model to produce the same answer it would have produced if the whole request had been in the system message. For conflicting requests they train the model to answer as if it had never seen the lower instruction. On closed tasks such as "summarise this," they argue there is no aligned user instruction: the model should summarise whatever is in the data slot. They recommend putting the task in the system message and the third-party text in a separate user message, so the two are not one concatenated string. For browsing, their version of the hierarchy treats every instruction found in a page or a tool result as misaligned. They generate those attacks with a red-team model rewarded for confusing the base model, then train the base model to predict the original answer. They deliberately omit jailbreak examples from training, and they omit prompt injection from tools other than browsing, so they can test whether the behaviour generalises.

Against a baseline fine-tuned only on capability data, they report higher robustness on every main safety evaluation they ran, including about a 63 percent improvement on system-prompt extraction and more than 30 percent on jailbreaks the training set did not contain. Capability metrics they list (TriviaQA, LAMBADA, HellaSwag) stayed comparable. The regression they do report is over-refusal: the model sometimes ignores or refuses a benign request. Those numbers are for that GPT-3.5 Turbo comparison. They are evidence that a privilege order can be learned and that it is not free of false refusals. They are not a claim about a model trained later.

Controls that do not rely on obedience

The following distinction is editorial: prompts, labels and model-based screening try to reduce misbehaviour; externally enforced permissions constrain effects even when the model proposes a bad action.

For a direct attack, where the user is the adversary, the page's layers are a lightweight classifier in front of the main model, a filter for known patterns, a system prompt that states how to refuse, and a response to people who keep trying, including throttling. The screen they describe uses structured output so the verdict is a boolean, and it says to treat a refusal from the screen model itself as a harmful verdict, because the screen can decline to look. Pattern filters are the layer Greshake et al. discuss how an attacker can route around. Use them as a cheap reduction of known strings, not as the boundary.

For an indirect attack the page's structural rules are:

The page also says to red-team the workflow with documents and tool outputs that contain injection attempts before you deploy, and to watch production outputs afterwards. For computer use and browser use it says Anthropic runs extra classifiers on screenshots and page text. That is a vendor control on those tools, not something you inherit by using a chat API.

Both approaches separate task and source content, but they use different channels: Wallace et al. place source text in a user message, while Anthropic recommends a tool result and explicitly rejects plain user-text placement for third-party content. Anthropic says JSON escaping prevents syntactic breakout and marks data unambiguously. That is a syntax statement, not proof the model cannot obey the text. Greshake et al. already showed that a model can be told, in natural language, to decode a payload. A JSON string is still text the model reads. The control's value is that your application, not the model, decides whether a tool runs.

That last point is the one that holds when every prompt defence fails. If a tool can send mail, move money, or read a secret, the authorisation has to be outside the model: an externally checked permission scope; our additional implementation recommendations are operation-scoped idempotency and informed confirmation for high-impact actions, and no credential sitting in the system prompt where a leaking attack (Perez and Ribeiro's second goal) can ask for it. Wallace et al. note that the system message may itself hold business logic and secrets, and that knowing the prompt helps later attacks. Anthropic's least-privilege guidance supplies the simpler rule: do not give the model access to secrets it does not need.

What to choose and why

Our recommendation: write the test before the prompt. An invariance test can check the boundary directly. A document that contains an instruction to change the task, call a tool, or recite the system text must leave the tool uncalled and the secret unprinted. If the only evidence of safety is that the model said it would not comply, the test has not run.

Later research makes the evaluation requirement sharper. Nasr et al. (2025) adapt attacks to twelve recent defences and report success above 90% for most. That result concerns their tested defences and budgets, not every possible system. Debenedetti et al.'s CaMeL separates trusted control flow from untrusted data and enforces explicit capabilities/policies. It reports solving 77% of AgentDojo tasks versus 84% undefended. Its threat model trusts the user prompt and memory; it does not prevent every misleading text answer or phishing message, and its interpreter/policies remain trusted components. These results support testing enforcement and its utility cost, rather than assuming stronger wording solves injection.

Risks and limits

• Perez and Ribeiro's rates are for text-davinci-002 and 35 public prompts. They show that small prompt choices move the rate. They do not rank current models.
• Greshake et al. argue that a complete defence was not available when they wrote, and their discussion of filters, RLHF, and supervisor models is a list of gaps. Later training, including the instruction hierarchy, changes the rates. It does not retire the indirect channel.
• Wallace et al. improved a GPT-3.5 Turbo baseline on their suite and recorded over-refusals. Generalisation beyond browsing was a test, not a guarantee for every tool you attach.
• Anthropic's page describes training and product classifiers for Claude, including Haiku 5.5 as the example screen model on the day the page was opened. Those claims are the vendor's. A screen model can itself be induced to refuse or to miss an encoded payload, which is the dilemma Greshake et al. already named.
• Pattern lists and delimiter tricks are not a security boundary.
• This article does not claim any NetLeverage product behaviour, and it does not provide attack strings.

Sources

Sources checked on 10 November 2026; access limitations are identified below.

1. Perez and Ribeiro, Ignore Previous Prompt: Attack Techniques For Language Models, arXiv:2211.09527 — https://arxiv.org/abs/2211.09527. Goal hijacking, prompt leaking, factors on text-davinci-002.
2. Greshake et al., Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection, arXiv:2302.12173 — https://arxiv.org/abs/2302.12173. Indirect delivery, threat taxonomy, limits of filters and supervisor models.
3. Wallace et al., The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions, arXiv:2404.13208 — https://arxiv.org/abs/2404.13208. Privilege order, aligned versus misaligned instructions, GPT-3.5 Turbo results, over-refusal.

4. Anthropic, Mitigate jailbreaks and prompt injections — https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/mitigate-jailbreaks. Direct versus indirect threat models, tool-result channel, least privilege, output screening.

5. Nasr et al., The Attacker Moves Second, arXiv:2510.09023v1 (10 October 2025) — https://arxiv.org/html/2510.09023. Adaptive evaluation of twelve defences.

6. Debenedetti et al., Defeating Prompt Injections by Design, CaMeL — https://arxiv.org/html/2503.18813. Capability enforcement, AgentDojo results, trusted-input assumptions and explicit non-goals.

This article was originally published on 10 November 2026.