MODULE 10 OF 12 · RELEASE 3

Agent Security & Excessive Agency

Assuming someone will trick your AI, and making sure that is not enough to hurt you.

Con artists do not break into buildings. They walk in wearing a high-visibility jacket and ask someone to hold the door. The defence was never a better lock — it was a rule that says visitors sign in regardless of what they are wearing.

So what is this really about?

People will hide instructions in anything your AI reads: a support message, a web page, a document. Sooner or later one will work. So the question is what happens then — and the answer should be "not much", because the AI never had the permissions to do real damage in the first place.

Instructions come only from your system; content from outside is kept structurally separate so it cannot change behaviour.

QUICK CHECK

Someone successfully tricks your AI into trying to delete every customer record. Why does nothing happen?

The words you will hear

Threat model: A written list of what is worth stealing, who might try, and how they would get in.
Excessive agency: An AI able to do far more than its job requires. The intern with the master key.
Least privilege: Only the access needed for the actual task, and no more.
Injection corpus: A collection of known trick attempts, run automatically so you notice when one starts working.

By the end you will be able to

Write down what could go wrong before it does.
Keep outside content from ever becoming an instruction.
Give every ability the narrowest access that still works.

Going deeper

Start from the assumption that it works

Most security thinking about AI systems tries to stop the model being manipulated. Better instructions, a classifier on the input, a stronger model.

All of that is worth doing and none of it is a foundation, because you are defending a boundary that is fundamentally blurry: instructions and content arrive as the same kind of thing, and the attacker writes half of it.

So invert the assumption. Design as though every manipulation succeeds, and ask what the attacker gets. If the answer is "an error message and a log entry", the system is sound. If the answer is "the customer database", no amount of prompt engineering fixes that.

Write the threat model down

It does not need to be long. Four columns:

What is worth taking?      customer data, credentials, money
Who might try?             a user, an insider, an automated scan
Where do they get in?      any text the model reads
Worst credible outcome?    per asset, stated plainly

"Any text the model reads" is usually a longer list than people expect: support tickets, uploaded documents, web pages you fetch, previous conversation, tool results, and the descriptions of external tools. Each is a place instructions can be hidden.

The exercise is valuable precisely because it is boring. Half the entry points get found while writing the list, not during an incident.

Keep untrusted content structurally separate

If instructions and content are concatenated into one string, nothing downstream can tell them apart — including the model.

Keep them apart structurally instead. System instructions come from your code only. Untrusted content goes in clearly delimited places, labelled as data. Tool results are data. Retrieved documents are data.

This does not make injection impossible; models can still be influenced by convincing text in a content slot. It substantially raises the cost, and — more importantly — it gives you something to test. You can assert that user content never reaches the system instruction position. You cannot assert that a model was sufficiently sceptical.

Excessive agency is the real vulnerability

An agent with a database connection that can run any query is a problem waiting for a trigger. Not because the model is malicious, but because the capability exists and something will eventually reach for it.

For each capability, ask what it would need to do its actual job — then grant exactly that. An agent that answers questions about one customer’s orders needs to read that customer’s orders. Not the orders table. Not the database.

This is the difference between a bad day and a breach. Same successful manipulation, radically different outcome, and the deciding factor is a scope decision made calmly in advance.

The corpus makes it a regression test

Collect attacks: public techniques, plus ones written specifically against your system and its tools. Run them in CI with an ordinary assertion — the destructive action does not happen.

The payoff is not today. It is the day someone restructures a prompt eighteen months from now and three of those tests go red before the change reaches production. Without the corpus, that change ships and nobody knows until it matters.

Contain automatically

Some patterns are worth stopping without waiting for a human: the same tool called fifty times in a minute, a sudden burst of refused calls, one actor touching far more records than anyone normally does.

Pause the actor and alert. A false positive costs someone a few minutes of inconvenience. A missed true positive costs you the incident. The asymmetry is obvious once stated, and it is the reason automatic containment beats an alert that fires at 3am into an empty room.

What good looks like

Someone can successfully convince your model to do something terrible, and the result is a logged refusal. You can say what each capability may reach without looking it up. And your injection corpus is a normal, boring part of CI.

How it is put together

Every input that reaches a prompt is classified by trust level, and untrusted content is never able to alter instructions. Permissions are scoped per actor and per tool, always at the narrowest workable scope. A standing injection corpus runs in CI. Anomalous tool-use patterns trigger containment rather than an after-the-fact alert.

Where you start

The tool registry from Module 3 with no trust classification on inputs.

The lab

Write the threat model: assets, actors, entry points and the worst credible outcome for each.
Classify every prompt input by trust level and keep untrusted content structurally separate from instructions.
Build an injection corpus from public techniques and your own system’s specifics.
Narrow every tool permission to the minimum that still works.
Add rate and pattern detection on tool use, with automatic containment.

What goes wrong, on purpose

The scenario. A retrieved document contains instructions telling the agent to exfiltrate data through a permitted tool.

What should happen. The instruction is treated as data, the tool’s permission scope does not allow the destination, and the attempt is recorded and contained.

How you prove it works

The injection corpus running in CI, failing the build on any successful injection.
A cross-tenant access test asserting no actor can reach another’s data.
A secret-exposure scan over logs, traces and model inputs.

What you walk away with

A written threat model, an injection corpus in CI, least-privilege tool scopes, and containment that triggers without a human.