EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

artefact
A saved file a release needs, such as the model, the code that prepares its inputs or the exact package versions.
batch
A run that scores many rows at once on a schedule and writes the results to a table.
canary
A release shown to a small random slice of traffic first, so a bad model reaches few users.
drift
A change in the inputs a model sees, or in the link between those inputs and the outcome it predicts.
evaluation gate
A check before release that stops a model when its measured quality falls below a set bar.
feature contract
An agreed promise about one input column — its type, range, missing values, uniqueness, freshness and owner — checked as data arrives.
incident
An event that degrades a service beyond its agreed level and needs a coordinated response.
inference contract
The agreed inputs, outputs and error cases for a model's prediction service, written down and tested.
latency budget
The time a request may spend in the model path, after network and rendering are subtracted from what users will tolerate.
model card
A short document describing what a model is for, how it was trained, its limits and who owns it.
monitoring
Watching inputs, outputs, service health and later outcomes so a change is noticed before users report it.
rollback
Returning to the last known good version, using a saved copy and a command that has been rehearsed.
serving
Running a model to answer requests in production, either one at a time or in scheduled groups.
training-serving skew
When an input is computed one way for model fitting and another way at request time, so the model reads different data.

MLOps from the ground up

Capstone: a release checklist with a rollback plan

A release checklist that runs: pre-flight checks, a canary with a decision rule, a promotion and a rollback drill with a timeline.

Part 10 of 10

Advanced18 min readBack to Courses

The checklist is code

A checklist in a document is followed until the first urgent release. A checklist as data — items, owners, required evidence — can be run, versioned and attached to the release. The script refuses to promote when a required item is missing, which is the only property that matters at 2am.

The checklist is also a contract between teams. The model owner supplies the evaluation, the platform owner supplies the artefact, the data owner supplies the schema, and the on-call engineer supplies the rollback. Each item names a person or a team, not a hope.

Version the checklist with the release and store the completed copy next to the artefact. The record answers the question every incident asks: what was checked, by whom, and what evidence was attached. A checklist that is overwritten each time cannot answer it. The checklist should also name what is out of scope, so a release does not quietly include a migration nobody reviewed.

Pre-flight

The artefact loads, the schema matches, the evaluation gate passed, the rollback bundle exists, the dashboard is live and the on-call knows. Each item has an owner and a link to evidence, not a checkbox and a promise. A pre-flight that can pass with an empty evidence field is decoration.

Run the pre-flight against the exact artefact that will serve traffic. A check against a rebuild is a check of a different thing, and the difference is exactly where release bugs live.

Automate what can be automated and sign what cannot. Loading the artefact, comparing the schema and calling the health endpoint are scriptable. “The on-call knows” is a human acknowledgement, and it should be a named person with a timestamp, not a box that anyone can tick.

Archive the bad version, restore the good one

from mlflow.tracking import MlflowClient

client = MlflowClient()
client.transition_model_version_stage(name="recommender", version=8, stage="Archived")
client.transition_model_version_stage(name="recommender", version=7, stage="Production")
print(client.get_model_version("recommender", 7).current_stage)

A checklist that blocks

Run the pre-flight and list every blocked item. Then fill in the missing evidence and mark the evaluation as passed, and confirm the release becomes ready without touching the other items.

Output
Click "Run" to execute your code

Canary and promote

A small slice, a fixed window, a pre-registered metric and a decision rule. Promote only when the canary beats the control within the tolerance; otherwise roll back and keep the evidence. The rule is written before the numbers arrive because the numbers are very persuasive after a long week.

Promotion is not the end of the release. The first thirty minutes at full traffic see a different mix from the canary, especially if the slice was not random. Keep the dashboard open, keep the rollback warm, and treat the post-promotion window as part of the test.

The canary slice should be random and large enough to see the segment that worries you. If the last two incidents came from a specific device or country, stratify the slice so that segment is represented. A canary that samples the easiest traffic is a rehearsal, not a test. A canary that nobody is watching is a slow rollout.

A rollback plan that has never been run is a wish. The drill is the only evidence that the old artefact loads, the command works and the time to healthy is what the document claims.

Rollback drill

Run the rollback before you need it. Measure the time from decision to healthy, and record it in the checklist. If the drill takes twenty minutes, the on-call can decide whether that is acceptable before an incident rather than during one.

Then write the postmortem: the trigger, the timeline, the user impact and the check that was missing. A rollback is a successful safety action. The failure is the gap that let the bad version reach traffic, and that gap is what the next checklist item should close.

Drill on a schedule, not only after an incident. Put it in the on-call rotation, record the time to healthy, and compare it with the last drill. A rollback that gets slower every quarter is a warning that the system is accumulating dependencies the plan does not cover. A drill that only the platform team can run is a single point of failure. Record the result in the release checklist.

Promote, observe, roll back

Run the release and read the timeline. Then raise ERROR_GATE to 0.02 and confirm the release survives the last observation, which is exactly the trade the gate encodes.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

Why make the release checklist executable?