Patternprompt-injectionai-agentsllm-security

Prompt Injection and the Abusable AI Surface

Shipping AI features creates attack surfaces classic appsec misses. Four forms of prompt injection and agent abuse seen in the wild, and the fixes.

Rahul Dharan·July 17, 2026·9 min read

What this is

Prompt injection is what happens when text a model reads as data gets followed as an instruction instead. Because a language model does not reliably separate the two, any attacker-controlled content that reaches its context can steer it, and shipping an AI feature wires that context up to real inputs and real outputs. The result is a set of attack surfaces classic application security does not cover: the model can be steered, its output can carry a payload, its sandbox can trust the wrong caller, and the endpoints behind it can go unauthenticated.

This is not the model “being jailbroken” in the chatbot sense. It is a systems problem. The vulnerability is rarely the model alone. It is how the model is wired to data, to the page, and to the tools around it.

Why it keeps happening

Classic appsec has a clean mental model: the client is untrusted, the server is trusted, and the boundary between them is where you check things. An AI feature breaks that model in two places at once. The model consumes untrusted input and produces output that gets treated as trusted, so the untrusted-trusted boundary now runs straight through a component that cannot be relied on to hold it.

Teams ship the feature reasoning about the happy path: a support assistant that answers questions, an agent that runs a helpful tool, a sandbox that renders generated code. The threat model that comes with it is unfamiliar, so it is often skipped. And there is a specific, comforting mistake that makes it worse: the assumption that a content safety classifier is the security control. Safety classifiers are tuned to refuse content categories. Prompt injection is a misuse class. A model tuned to never reveal “secrets” will still happily write a working cross-site scripting payload if you ask it to help format some HTML, because that output is not in any category the classifier was told to refuse. The safety layer is real and it is blind to this.

How the attack works

Four forms recur, and they are best understood as four different boundaries failing.

Indirect prompt injection into a retrieval assistant. A RAG or support assistant answers questions by retrieving internal content and feeding it to the model. An attacker crafts queries, or plants content the assistant will retrieve, that steer the model into revealing what is in its context: internal knowledge-base articles, administrator email addresses, fragments of credentials that were sitting in the retrieved documents. The user asked a question. The model answered a different one the injected text supplied.
The model emitting an attacker-controllable payload that is rendered unsafely. The AI writes output that the application then renders, and the output contains active content, a script tag, an event handler, a link. If the application renders model output without sanitizing it, the model has just written the attacker’s cross-site scripting payload for them. The safety classifier waved it through because injection is not a category it screens.
An agent sandbox that trusts its embedder. AI features often run generated code in a sandboxed iframe. The isolation only holds if the sandbox validates who is talking to it and refuses to run on the wrong origin. When the only protection is that the parent page remembered to set the right sandbox and frame-ancestors attributes, the control is defeated by omission: leave the attributes off and the sandbox executes attacker JavaScript on the real application origin.
Unauthenticated endpoints that mint agent credentials or run agent tools. Behind an AI feature sit endpoints that issue voice-agent or model credentials and invoke tools. When one of those endpoints ships without authentication, an anonymous caller can mint the credentials the agent uses or run the tools the agent runs. The AI is the front door and the door has no lock.

What it looks like in the wild

We have seen each of these forms in production, across companies that share nothing but having shipped an AI feature.

A payments company ran a RAG support assistant that answered from an internal knowledge base. Through indirect-injection queries, the assistant could be steered into returning internal knowledge-base content it should never surface to an outside user, including administrator email addresses and fragments of credentials that lived in the retrieved articles. The assistant was working exactly as built. It retrieved relevant internal content and summarized it. The injection just redefined “relevant.”
An AI GTM tool let the model generate content that was published to live pages, and the model could be made to emit raw script tags. Because the published pages served that output with no sanitization, the model’s output became stored cross-site scripting on real pages. The team had a safety classifier on the generation step. It screened for the content categories it knew about, and a script tag was not one of them.
An insurtech’s AI-assistant sandbox ran generated code in an iframe that validated neither the origin of the messages it received nor its own frame-ancestors. As a result it executed attacker-supplied JavaScript on the real application origin, with the access that origin carries. The isolation was entirely delegated to the parent remembering to set two attributes, and the failure mode was simply not setting them. The full teardown is in The Sandbox That Trusted Its Parent.
An AI recruiting tool exposed an unauthenticated route that minted voice-agent credentials. Anyone who found the route could obtain the credentials the product used to run its voice agent, no account required. The AI feature was the whole product, and the endpoint that powered it was open.

The consistent theme: in every case the AI feature did its job. The abuse came from the boundary around it, not from the model misbehaving in isolation.

How to tell if you are exposed

A self-check for any team that has shipped an AI feature:

Trace where model output gets rendered. Follow every path from model output to a rendered page, a document, or another system. At each one, ask whether the output is sanitized and encoded before it is rendered. If model output reaches a page as HTML without sanitization, you have a stored-XSS primitive that the model can be steered to fill.
Try to steer your own retrieval assistant. Feed your RAG or support assistant queries and planted content designed to make it reveal what is in its context. If it will surface internal articles, addresses, or credential fragments that sit in its retrievable corpus, its context is its blast radius.
Check your sandbox for origin and frame-ancestors validation. If you run generated code in an iframe, confirm it validates the origin of inbound messages and restricts frame-ancestors, and that neither is left to a single attribute on the parent. Remove the parent’s attributes in a test and confirm the sandbox still refuses to run.
Enumerate the endpoints behind the AI feature. List every route that mints an agent or model credential or runs an agent tool, and send each an unauthenticated request. Any that responds is open.
Ask what the model can reach. Inventory the secrets and tools that are in the model’s context or callable by it. Anything reachable is anything an injection can reach.

How to fix it

The durable posture is to assume the model will be injected and contain what that can touch.

Treat all model output as untrusted input. Sanitize and context-encode it before it is rendered anywhere, exactly as you would treat a raw request body. The model is an untrusted producer, not a trusted one.
Isolate agent sandboxes, and validate message origin AND frame-ancestors. Do not delegate isolation to a single attribute on the parent. Validate the origin of every message the sandbox receives, restrict who may frame it, and fail closed if either check is missing.
Keep secrets and privileged tools out of the model’s reachable context. Assume prompt injection is possible, so the only safe secret is one the model cannot reach. Retrieval corpora should not contain credentials, and tool access should be scoped so a steered model cannot escalate.
Authenticate every AI and agent endpoint. Every route that mints a credential or runs a tool needs real authentication, the same as any privileged API. The AI feature does not exempt the endpoint behind it.
Red-team the AI surface specifically. A content safety classifier is not an authorization control and does not screen for injection. Test the AI surface as its own attack surface: injection, output handling, sandbox isolation, and the endpoints behind it, not just the classic API.

The takeaway

Assume the model will be injected, and make sure that when it is, there is nothing dangerous within its reach. Treat its output as untrusted, isolate its sandbox, authenticate its endpoints, and red-team the AI surface as its own attack surface, because a safety filter tuned to refuse secrets is blind to the injection that steers everything around it.

Frequently asked

What is prompt injection?

Prompt injection is an attack where text the model reads as data is instead followed as an instruction. In indirect prompt injection, the malicious text arrives through content the model retrieves, such as a document, a support-knowledge-base article, or a web page, rather than from the user directly. Because a language model does not reliably separate instructions from data, attacker-controlled content in its context can steer its behavior, leak its context, or make it emit attacker-chosen output.

Why does shipping an AI feature create new attack surface?

An AI feature turns untrusted input into model behavior and model output into rendered content, two boundaries classic application security does not police. The model can be steered by injected instructions, its output can carry an attacker's payload into your page, its sandbox can trust a caller it should not, and the endpoints that mint agent credentials or run agent tools may never have been authenticated. None of these are caught by testing the classic API alone.

How do you prevent prompt injection?

You cannot fully prevent a model from being injected, so you contain the blast radius. Keep secrets and privileged tools out of the model's reachable context, treat all model output as untrusted and sanitize it before rendering, isolate agent sandboxes and validate both message origin and frame-ancestors, authenticate every endpoint that mints agent credentials or runs agent tools, and red-team the AI surface specifically rather than assuming a safety classifier covers it.

Can a content safety filter stop prompt injection?

Not reliably. Safety classifiers are usually tuned to refuse categories like secrets, violence, or hate speech. Prompt injection is a misuse class, not a content category, so a model can be steered into emitting a working XSS payload or leaking retrieved context while every output still passes the safety filter. The filter is blind to the class of abuse it was not trained to see.

This is one finding from a harness that runs continuously. See how Greywatch finds, proves, and fixes them.

How it works →