Threat model for an LLM feature

Assets, effects and blast radius

Start with what the feature can touch, not with the model. A support assistant can read customer records, write a reply, issue a refund and spend tokens; a coding assistant can read a repository, edit a file and run a command. Those are the assets and the effects, and they exist whether the model is a transformer or a lookup table.

For each effect, write the blast radius: one user or every tenant, reversible or not, who notices and how quickly. Drafting an email and sending it are different products. The blast radius is what decides how much control the effect needs, and it is the first thing a reviewer should ask about.

Write the asset list in nouns that a lawyer and an engineer both recognise: the customer record, the refund, the repository, the model context. A list of abstractions such as data or trust cannot be reviewed, because nobody can point at it or say whether it is in scope.

Actors, entry points, trust boundaries

The actors are the people and systems that can influence the feature: the user, another tenant, a retrieved document, a tool result, the author of a plugin, an insider with log access. The entry points are where their text arrives: the input box, an upload, a fetched page, a webhook, a previous turn. List them; the list is short and it is usually incomplete.

Then draw the boundary. The model sits on it, not inside the trusted zone. Everything it reads may be attacker-controlled, and everything it emits is untrusted until a check says otherwise. That one sentence reorganises the design, because it puts the model in the same category as a browser request rather than the same category as your own code.

A trust boundary is only useful if something crosses it. Mark the crossings: a document entering the prompt, a model output entering a template, a tool result entering a decision. Each crossing is a place where a check can live, and a boundary with no crossing is a diagram.

A threat as data

from typing import Literal
from pydantic import BaseModel, Field


class Threat(BaseModel):
    asset: Literal["customer_record", "refund", "source_code", "context"]
    entry: Literal["user", "document", "tool_result", "plugin"]
    effect: Literal["read", "write", "send", "spend"]
    reversible: bool
    tenants: Literal["one", "many"]
    control: str = Field(min_length=4)


THREATS = [
    Threat(asset="customer_record", entry="document", effect="send",
           reversible=False, tenants="one", control="provenance check before send"),
    Threat(asset="refund", entry="tool_result", effect="spend",
           reversible=False, tenants="many", control="amount cap and human approval"),
]

Likelihood times impact, roughly

You are ordering work, not doing actuarial maths. Give impact a small integer from one to three and reachability the same, then multiply. A remote path that any user can take outranks a theoretical one that needs a compromised insider, and the product makes that ordering explicit enough to argue about.

Re-score after each control. A control that only logs does not reduce the score, because it changes what you know afterwards, not what can happen. A control that blocks, caps or requires approval does, and the new score is what tells you whether the next control is worth building.

The score is not a promise of safety; it is a queue. Two people scoring the same feature will disagree by a point, and that is fine as long as the disagreement is about impact or reach rather than about whether the effect exists at all.

Keep it alive

A threat model is a file in the repository, reviewed when a tool, a data source or a permission is added. The tool list is the attack surface, so a diff to the tool list is a diff to the model. If the review is a document nobody opens, it is a compliance artefact rather than a design tool.

After an incident, add the case to the red-team suite in chapter six and re-score. The model is a living list, not a certificate, and the version that matters is the one that matches the code on main.

Assign an owner. A threat model with no owner is updated by nobody, and the first sign is a tool that appeared in production without a line in the document. The owner reviews the diff that adds it, the way they would review a new permission.