EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

artefact
A saved file a release needs, such as the model, the code that prepares its inputs or the exact package versions.
batch
A run that scores many rows at once on a schedule and writes the results to a table.
canary
A release shown to a small random slice of traffic first, so a bad model reaches few users.
drift
A change in the inputs a model sees, or in the link between those inputs and the outcome it predicts.
evaluation gate
A check before release that stops a model when its measured quality falls below a set bar.
feature contract
An agreed promise about one input column — its type, range, missing values, uniqueness, freshness and owner — checked as data arrives.
incident
An event that degrades a service beyond its agreed level and needs a coordinated response.
inference contract
The agreed inputs, outputs and error cases for a model's prediction service, written down and tested.
latency budget
The time a request may spend in the model path, after network and rendering are subtracted from what users will tolerate.
model card
A short document describing what a model is for, how it was trained, its limits and who owns it.
monitoring
Watching inputs, outputs, service health and later outcomes so a change is noticed before users report it.
rollback
Returning to the last known good version, using a saved copy and a command that has been rehearsed.
serving
Running a model to answer requests in production, either one at a time or in scheduled groups.
training-serving skew
When an input is computed one way for model fitting and another way at request time, so the model reads different data.

MLOps from the ground up

Rollback and canaries

A release is not finished when it deploys. It is finished when it can be undone, and the undo has been tested.

Part 5 of 10

Intermediate17 min readBack to Courses

Four ways to release

Shadow sends traffic to the new model without showing its output. Canary shows it to a small slice. Blue-green switches all traffic between two identical environments. A feature flag chooses between models per request. They differ in blast radius and in how quickly you can undo.

Most teams need two of them: a canary for the model and a flag for the code path that calls it. The canary limits how many users see a bad model; the flag lets you change the routing without a deploy. Together they turn a release into a sequence of small, reversible decisions.

Blast radius is the variable to compare. Shadow has none, a canary has a slice, blue-green has all traffic at the moment of the switch, and a flag has whatever the flag selects. Choose the smallest blast radius that can still answer the question the release is asking. The pattern is not a maturity ladder; a team with one server and a flag can release more safely than a team with a canary nobody watches. Choose the one you can operate.

Design the canary

Choose the slice, the metrics, the duration and the decision rule before the release. A canary that watches everything watches nothing. Compare against the control on the same window, and pre-register the threshold that triggers a rollback, so the decision is not made by whoever is most optimistic in the room.

The slice should be random, not the first users who arrive. Early traffic has a different mix, and a canary that only sees the morning crowd is not a sample of the day. If the product has segments that matter, check that the slice contains them.

Sample size decides whether the canary can answer at all. A five percent slice of a thousand daily users gives fifty people, which is enough to catch a crash and not enough to detect a two percent conversion lift. Calculate the detectable effect before the release, or the canary will end in a shrug.

Move a model version in the registry

from mlflow.tracking import MlflowClient

client = MlflowClient()
client.transition_model_version_stage(
    name="recommender",
    version=7,
    stage="Production",
    archive_existing_versions=True,
)
print(client.get_model_version("recommender", 7).current_stage)

Promote, extend or roll back

Run the three scenarios and explain each decision. Then raise MAX_ERROR_RATE to 0.05 and confirm the third scenario changes from a rollback to a promote.

Output
Click "Run" to execute your code

Rollback is an artefact

The previous model, its preprocessing and its environment must still exist and still be loadable. A rollback that requires a rebuild is a second release under pressure. Keep the last known good bundle and the command that swaps it, and run that command in a drill.

The drill is what separates a plan from a hope. Measure the time from decision to healthy and write it down. If the number is twenty minutes, the on-call can decide whether that is acceptable before an incident rather than during one.

The rollback command should be one command, owned by one team, and tested on a schedule. If it needs a database migration reversed, a cache cleared and a queue drained, those steps belong in the script, not in someone’s memory. The script is the plan; the document is the explanation.

A rollback that requires a rebuild is not a rollback. If the old artefact is gone, the fastest safe path may be to turn the feature off, which is a different plan and needs its own drill.

After the rollback

Write down the trigger, the timeline and the user impact. A rollback is a successful safety action, not a failure to hide. The postmortem asks which signal was late and which check would have caught it earlier, not who pressed the button.

Then fix the check, not just the model. A canary that missed the regression is a monitoring gap, and the next release will meet the same gap with a different model. The incident is only useful if it changes the system that allowed it.

Communication is part of the release. Tell the support team what changed, tell the stakeholders what was rolled back, and say when the next attempt will be. A silent rollback teaches users that the product is unreliable and teaches the team that incidents are private.

A release with an undo

Run the timeline, then comment out the whole recovery block and confirm the service stays on the unhealthy version. The plan is only real when the undo is executed.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What does a shadow release do?