EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

artefact
A saved file a release needs, such as the model, the code that prepares its inputs or the exact package versions.
batch
A run that scores many rows at once on a schedule and writes the results to a table.
canary
A release shown to a small random slice of traffic first, so a bad model reaches few users.
drift
A change in the inputs a model sees, or in the link between those inputs and the outcome it predicts.
evaluation gate
A check before release that stops a model when its measured quality falls below a set bar.
feature contract
An agreed promise about one input column — its type, range, missing values, uniqueness, freshness and owner — checked as data arrives.
incident
An event that degrades a service beyond its agreed level and needs a coordinated response.
inference contract
The agreed inputs, outputs and error cases for a model's prediction service, written down and tested.
latency budget
The time a request may spend in the model path, after network and rendering are subtracted from what users will tolerate.
model card
A short document describing what a model is for, how it was trained, its limits and who owns it.
monitoring
Watching inputs, outputs, service health and later outcomes so a change is noticed before users report it.
rollback
Returning to the last known good version, using a saved copy and a command that has been rehearsed.
serving
Running a model to answer requests in production, either one at a time or in scheduled groups.
training-serving skew
When an input is computed one way for model fitting and another way at request time, so the model reads different data.

MLOps from the ground up

Incident playbook

An incident is a decision problem under time pressure. Severity, roles, mitigation before diagnosis and a blameless review are what turn a bad hour into a better system.

Part 9 of 10

Intermediate17 min readBack to Courses

Severity is a decision, not a feeling

An incident is an event that degrades a service beyond its agreed level. The first job is to assign a severity, because severity decides who is paged, how often you communicate and who can authorise a rollback. Without a definition, every incident is either an emergency or nothing, and both are expensive.

A workable scale has three or four levels. The top level is user-facing loss with no workaround; the next is degraded service with a workaround; the next is a risk that has not reached users yet; the last is a question for the next business day. Each level names a response time, a communication channel and an owner.

Severity can change. Start high when the impact is unknown and downgrade once the scope is clear. Starting low and escalating after twenty minutes of confusion is the more common mistake, and it costs the exact minutes that matter. The cost of an unnecessary page is a conversation; the cost of a late one is the incident.

Roles, not a group chat

An incident needs an incident commander, a communications lead and one or more people working the system. The commander does not debug; they keep the timeline, make decisions and stop the team from converging on the same hypothesis. The communications lead owns the status page and the updates, so the engineers can stay in the logs.

In a small team one person can hold two roles, but the roles still have to be named. The failure mode is a channel where everyone is investigating and nobody is deciding. Write the names at the top of the incident document, and hand them over explicitly when someone leaves.

The commander’s first three questions are: what is the user impact, what changed recently, and what is the fastest way to reduce the impact. The third question is the one that separates an incident response from a debugging session. Root cause can wait until users are no longer affected.

The metrics an incident needs

from prometheus_client import Counter, Histogram

requests = Counter("inference_requests_total", "Requests served", ["status"])
latency = Histogram("inference_latency_seconds", "Request latency")

with latency.time():
    requests.labels(status="ok").inc()

Classify and page

Run the classifier and check which incidents page a human. Then add a data-loss flag to the batch-job incident and confirm it escalates to the top level.

Output
Click "Run" to execute your code

Mitigate before you explain

The fastest mitigation is usually to undo the last change. Roll back the deployment, disable the feature flag, shift traffic to the healthy region, or turn off the expensive path. None of those requires knowing why the change broke, and all of them stop the bleeding. Diagnosis continues on a copy of the bad state, not on the live system.

Feature flags are the cheapest rollback because they do not need a deployment. That only holds if the flag is wired to a safe default and someone has tested the off position. A flag that has never been turned off in staging is a rollback plan with a syntax error.

Communicate on a cadence, even when there is nothing new. A short update every thirty minutes with the impact, the current action and the next update time is more useful than a detailed explanation after the fact. Silence is read as incompetence, and it invites a second channel where people speculate.

A rollback that has never been exercised is a hypothesis. The incident is the worst time to discover that the old image no longer starts or that the flag has no off switch.

The postmortem and the next incident

A blameless postmortem describes what happened, what was detected, what was tried and what actually worked. It is not a search for a person; it is a search for the missing check, the ambiguous alert, the rollback that took too long or the dashboard that pointed the wrong way. The output is a small set of changes with owners and dates.

Track the time to detect and the time to mitigate separately. A fast mitigation with a slow detection is an alerting problem; a fast detection with a slow mitigation is a tooling or authority problem. Averaging them into one number hides which one to fix.

Rehearse the response before it is needed. A game day that injects a dependency failure, a bad deploy or a full queue finds the runbook that is out of date and the person who does not have permission. The drill is not a test of the team; it is a test of the system the team will use at three in the morning.

Detection, mitigation, communication

Run the incident and compare the time to detect with the time to mitigate. Then move the alert to minute 6 and see which number changes.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

Why assign a severity at the start of an incident?