Incident response

An incident is a decision under uncertainty

An LLM feature fails in ways the dashboard does not name: a refusal rate that doubles, a prompt injection that reaches a tool, a model that starts leaking a cached document, a provider that deprecates the model you pinned. The first job is not to explain it. It is to decide who owns it and how fast they must move.

Write the runbook before the incident, because the person on call at three in the morning should not be inventing a process. The runbook names the signals, the first three checks, the containment levers, the escalation path and the person who can approve a shutdown. A runbook that requires its author is not a runbook.

The hardest part is often agreeing that it is an incident. A single bad answer is a bug; a pattern of bad answers on a regulated topic is an incident. The threshold belongs in the policy, with examples, so the decision does not depend on who happens to be awake.

Severity sets the response

Severity is impact times scope, not the loudness of the reporter. A wrong answer about a refund is annoying; a wrong answer that discloses another customer’s data is a breach. A prompt injection that only changes a summary is contained; one that reaches a refund tool has already had an effect.

Define four levels and what each one buys: who is paged, how often updates go out, whether the feature is disabled, whether legal or security is involved. The definitions should be concrete enough that two engineers classify the same event the same way. A severity table with no examples is a mood.

Severity can rise. Start at the level the evidence supports and raise it when scope grows; do not wait for certainty before paging. The cost of an unnecessary page is an hour of attention, while the cost of a late breach notification is measured in regulators.

Open the channel from the runbook

import os
from slack_sdk import WebClient

client = WebClient(token=os.environ["SLACK_BOT_TOKEN"])
channel = client.conversations_create(name="inc-2026-10-30-prompt-leak")["channel"]["id"]
client.chat_postMessage(
    channel=channel,
    text="SEV-2 opened: prompt leak in support assistant. IC: @ada. Runbook: /runbooks/prompt-leak",
)

Contain first, explain later

The levers are few and should be named in advance: disable the tool, roll back the prompt or policy version, block the tenant, rotate the key, switch the provider, turn off prompt capture. Each one has a cost, and the runbook should say what it is so the on-call can choose without a meeting.

Preserve evidence while you contain. The trace, the prompt version, the tool arguments and the policy record are what the review will need, and a rollback can overwrite all three. Copy them to an incident bucket before the fix, and record the time of the copy. Evidence collected after the change explains a system that no longer exists.

Containment is not a fix. Disabling a tool stops the bleeding and leaves the product broken, which is the right trade for an hour and the wrong one for a week. Set a time to revisit the containment and name the person who owns the real fix.

The review is the deliverable

A postmortem is a timeline, a set of contributing factors and a short list of changes with owners and dates. It is not a search for a person to blame; blame makes the next incident quieter and later. The question is what made the wrong action easy and the right one hard.

Build the timeline from timestamps, not memory: first signal, first acknowledgement, containment, resolution, customer impact. The gaps between them are the metrics that improve the next response. A timeline written a week later from recollection is a story.

Every action item needs an owner and a date, or it is a wish. Prefer the change that removes a class of failure — a policy default, a missing check, a tool that cannot be called without approval — over the one that fixes a single string. Then run a tabletop exercise on the runbook you just changed.