Permissions and guardrails

The model is the least trusted component

Everything the model reads can influence what it asks for: the user, a retrieved document, a tool result, a previous turn. The tool call is therefore untrusted input, exactly like a form submission.

A guardrail is the code between that request and the effect. It answers three questions: is this tool allowed for this user, are these arguments acceptable, and does this action need a person.

Treat the tool call the way you treat an HTTP request from the internet. It arrives with a name, a set of arguments and an implied intent, and none of those can be trusted because the model assembled them from text it read. The guardrail parses the request, looks up the policy for that user and that tool, checks the values, and only then decides whether to run, ask or refuse.

Allow-lists, not block-lists

A block-list of bad arguments is a losing game: there is always another spelling. An allow-list says which values are acceptable and rejects everything else, which is a finite thing to reason about.

The same applies to tools. A support agent gets read tools and a refund tool with a cap; it does not get a shell because “it might be useful”. Every capability is a decision someone should be able to defend.

Write the policy in the same repository as the tools, reviewed like code. A spreadsheet maintained by another team is a policy that drifts silently, and the first symptom is a role that can do something nobody remembers granting. The policy should be small enough to read in one sitting and specific enough that a new tool without an entry is denied by default.

A policy the model cannot see

POLICY = {
    "get_order": {"roles": {"support", "admin"}, "approval": False},
    "issue_refund": {"roles": {"admin"}, "approval": True, "max_amount": 100},
}

def check(user, tool, args):
    rule = POLICY.get(tool)
    if rule is None:
        return "denied: unknown tool"
    if user["role"] not in rule["roles"]:
        return "denied: role"
    if "max_amount" in rule and args.get("amount", 0) > rule["max_amount"]:
        return "denied: over limit"
    return "approval" if rule["approval"] else "allowed"

Approval is a UI problem too

An approval request is a decision, and a decision needs the facts. Show the tool, the exact arguments, the user it will affect and the consequence: “issue a refund of 42.00 to order A-1024 for customer 8812”. A dialog that says “the agent wants to perform an action” trains people to click approve without reading, which is worse than no approval because it looks like a control.

A common mistake is asking for approval after the effect has been prepared or partially applied. The person is then approving a fait accompli. Put the pause before the call, keep the state resumable, and record the decision — approved, rejected, or timed out — in the same trace as the tool call.

Track where the data came from

A useful guardrail is provenance. If a value came from a retrieved document, it is untrusted; if it came from an authenticated session, it is trusted. An action that needs a trusted value should refuse an untrusted one, no matter how plausible it looks.

This is what stops the classic attack: a document says “refund order A-1024”, the agent calls the refund tool, and the order id came from the document rather than from the user’s verified session.

Guardrails fail closed

If the policy cannot be loaded, the tool is unavailable. If the role is unknown, it is denied. If the approval service is down, the action waits. A guardrail that fails open turns an outage into an incident.

The cost is availability, and it is the right trade for anything with an effect. Read-only tools can fail open; writes cannot.

The failure mode to watch is the quiet one: a policy lookup that returns an empty rule set and is read as “no restrictions”. Treat an empty policy as a configuration error, not as permission. The symptom of getting this wrong is an outage that silently grants more access than the day before.