The worst thing our agent could do was quiet. It could write a wrong value into records people rely on, across a whole project, and nobody would notice.
The prompt has nothing to do with what stops it. In the agentic version of a document platform we are building, the model has no tools at all. It returns one JSON object that must match a strict schema. Our code checks that object, and only our code writes, through the same internal API the rest of the product uses. Every model call is stored with what went in and what came out. When a person confirms a value, that decision wins over every later run.
We also built undo. Every run records the values it replaced, so a person can roll back a whole run. Review catches the mistakes people see. Undo fixes the ones they find later.
Everything below comes from that system and from the queue-based pipeline it sits on.
Permissions go in code
A prompt that says "don't delete anything" is a request. It is not a control. The tool decides what it allows. The database decides who can write what. The prompt only explains.
OWASP's 2025 list of risks for LLM applications calls this Excessive Agency and names three causes: too much functionality, too many permissions, too much autonomy. Its advice is the same rule in other words: enforce authorisation in the systems downstream of the model, rather than asking the model whether an action is allowed.
In practice:
- Functionality. Give a tool one job ("save this field on this record"), not a general one ("run this SQL"). Delete tools left over from experiments.
- Permissions. The database user behind a read tool can only read. Actions for a user run with that user's rights, not an admin account.
- Autonomy. Anything hard to reverse waits for a person, and the check lives in the tool, where the model can't talk its way past it.
Our extreme version of this is to give the model nothing to call. That won't fit every agent. But each tool you add is a permission you now have to defend, so add them one at a time.
Read every document as if a stranger wrote it
Often one did. An agent that reads PDFs, emails or web pages is reading text that someone outside your company controls. That text can contain instructions, and the model has no reliable way to tell your instructions from theirs. OWASP calls this indirect prompt injection and says plainly that it is unclear whether any fool-proof prevention exists. Greshake and colleagues showed it working against real products back in 2023.
So you design for the day the model is tricked. Simon Willison's lethal trifecta is the best single test I know. Does one agent have all three of these?
- Access to private data.
- Exposure to content an attacker could control.
- A way to send data out: an HTTP request, an email, an image link, even a link the user is asked to click.
If the answer is yes, an instruction hidden in a document can read your data and ship it out. Our document agent has the first two legs: client documents and scanned pages from outside. It has no third, because its only output is a JSON object that our code validates. A narrow schema also leaves an injected paragraph very little room: it has to survive being squeezed into a date or a fixed list of values.
The design patterns paper that Willison reviewed starts from one strict rule: once an agent has read untrusted input, that input must not be able to trigger anything consequential. Fix the plan and the recipients before the agent reads the untrusted text, and let the text change only the content.
Queues deliver at least once, so agents act at least once
An agent that runs on a queue will sometimes run the same step twice. Amazon SQS says so in its docs and asks you to make your handlers idempotent. If you haven't designed for it, a repeated step happens twice: two emails, two rows, two charges.
We found this in our own pipeline. A final "save the results" step called our API, and the 60-second response got lost. The retry ran the whole step again. An audit later found more than 10,000 surplus duplicate rows. The fix was a receipt: the step claims a row keyed by the run id first, inside the same transaction as its writes. If a completed receipt is already there, it replays the stored result and touches nothing.
For agents the rule is simple. Every tool that changes something takes an idempotency key, and checks "already done, skip" before it acts. The background-jobs version of this is in why AI features need background jobs.
Log every model call
Log what went in, what came out, which tool the model called with which arguments, and what came back. We needed it every single time a user said "this answer is wrong". The first question was always "what did the model see?". Without the log, you are guessing.
The same log is your bill. OWASP's Unbounded Consumption entry describes "denial of wallet": someone drives up your model spend on purpose. Put a limit on steps, tokens and time for every run, and you will see it in the log when a run hits it.
What I check first in someone else's agent
- What can it write, and who checks that?
- What does it read that someone outside could have written?
- Can it send anything out?
If the same agent reads untrusted text and can also send data out, that is the hole. Everything else is detail. This review is also where we start on agentic automation work for clients, and the agents at scale piece covers what changes when the same agent runs thousands of times a minute.
Do this week
Write your agent's tools in a table with three columns: reads private data, reads outside content, can send data out. Any agent with a mark in all three columns loses one before its next release.
The model will be tricked one day. Decide now what it is allowed to do on that day.