EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

accuracy
The share of predictions that are correct, counting every class the same.
additive
Built by summing separate contributions, so each part’s effect can be read on its own.
baseline
A reference score from a rule so plain it needs no learning, such as always naming the most common answer.
features
The input columns for one example — the numbers a model is allowed to read before it predicts.
fitting
Adjusting a model’s numbers so its predictions get closer to the answers in the training data.
generalisation
How well a model performs on examples it never saw while being fitted.
hypothesis class
The family of shapes a model is allowed to use, such as a straight line or a tree.
label
The correct answer for one example — what you want the model to predict.
loss
A single number measuring how wrong a prediction is; smaller means better.
metric
A single number used to judge predictions, such as the share answered correctly.
noise
Random variation in data that carries no real signal, which a model can mistake for a pattern.
optimiser
The procedure that changes a model’s numbers to make its loss smaller.
overfitting
When a model memorises accidental details of its training examples and does worse on new ones.
parameters
The numbers inside a model that fitting adjusts, as opposed to the inputs it reads.
regularisation
Any penalty or limit that discourages a model from chasing noise in its training examples. The aim is better performance on new data.
supervised learning
A setting where each example carries the answer you want the model to predict.
test set
Examples kept out of fitting, used to estimate how the model will do on data it has not seen.
training set
The examples a model is fitted on; a score there is not an estimate of performance on new data.
weights
The numbers a model multiplies each input by before adding the results together.

Classical machine learning

Capstone: an honest model card

Train a small classifier, evaluate it once on held-out data, and write the numbers a reader needs: dataset, split, metrics, subgroups, calibration and limits.

Part 10 of 10

Intermediate18 min readBack to Courses

A model card is a claim with evidence

A model card is a short document. It says what a model is for, what data it saw, how it was checked and where it fails. It is not a marketing page and not a leaderboard entry. Its job is to let a reader decide whether the model suits a use you did not anticipate. Most models eventually find themselves in that situation.

The evidence has to match the claim. “Accurate on customer churn” needs a definition of churn. It needs a time window, a split that respects time, and a baseline. “Works for everyone” needs results per group, not an overall average. A card that reports one number without its denominator or its group is a slogan with a table around it.

The evaluation plan

Write the plan before training. Name the unit of prediction, the label source, the split strategy, the baseline and the main metric. Decide which groups matter and what sample size each will have. Decide what counts as a failure worth blocking a release. These choices are the real model design. The algorithm is often the least consequential part.

Check the model once on the held-out set. Then report the confusion matrix, precision, recall and a view that does not fix a threshold, such as the precision-recall curve. Add calibration if the score will be read as a probability. Report the sample size next to every slice. Do not compare a group of twelve rows with one of twelve thousand as if the numbers carried the same weight.

The split should reflect how the model will be used. A random split is right for independent rows. A grouped split is right when rows share a customer. A time split is right when the future is the test. Getting this wrong is not a small bias. It can turn a model that looks ready to ship into one that has seen the answer during training.

The numbers a card needs

from sklearn.metrics import classification_report, confusion_matrix

print(confusion_matrix(y_test, predictions))
print(classification_report(y_test, predictions, digits=3))

Write the card

Run the evaluation, then add a second feature and a rule that uses both columns and rewrite the subgroup section with the new numbers.

Output
Click "Run" to execute your code

Honest means specific

A precise limitation is more useful than a vague disclaimer. “The model was trained on customers from one region in 2024 and has not been tested on new accounts” tells a reader exactly what to check. “Performance may vary” tells them nothing. The same applies to metrics. Name the threshold, the class balance and the date of the check.

Report the gap between training and test, not only the test score. A large gap is a warning about variance even if the test number is acceptable. Report the baseline beside the model. Name the group the score describes.

Report the metric you did not optimise as a guardrail. If recall improved because precision collapsed, the product may be unusable. The headline number still moved in the right direction. A guardrail is a second metric with a floor, agreed before the run. Then a trade cannot be presented as a pure win after the fact.

A single accuracy number hides the class balance, the threshold and the subgroup gaps. If the card contains one number and no denominator, it is not ready to be used for a decision.

What to write down

The minimum card has six parts. They are the intended use, the training data and label definition, the split strategy, the baseline, the metrics with sample sizes, and the known failure modes. Add the threshold and the calibration table if the output drives a decision. Add results per group if the population is not uniform. Add the date. A model card without a date is a claim about a moving target.

The card should also say what would change the answer. A new region, a new product line, a shift in the label definition or a change in the threshold all weaken part of the evidence. Naming those triggers turns the card from a snapshot into a maintenance document. That is what a deployed model actually needs.

Name the date of the check and the data version as well. A model is a function of both, and a score without them cannot be repeated. If the training data is refreshed monthly, the card has a shelf life. The next review should start by checking whether the population has moved. Only then should it check whether the code has changed.

Are the probabilities honest?

Bin the predictions, compare each bin’s mean score with its observed frequency, then split the last bin at 0.9 and see whether the expected calibration error changes.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What is the purpose of a model card?