EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

accuracy
The share of predictions that are correct, counting every class the same.
additive
Built by summing separate contributions, so each part’s effect can be read on its own.
baseline
A reference score from a rule so plain it needs no learning, such as always naming the most common answer.
features
The input columns for one example — the numbers a model is allowed to read before it predicts.
fitting
Adjusting a model’s numbers so its predictions get closer to the answers in the training data.
generalisation
How well a model performs on examples it never saw while being fitted.
hypothesis class
The family of shapes a model is allowed to use, such as a straight line or a tree.
label
The correct answer for one example — what you want the model to predict.
loss
A single number measuring how wrong a prediction is; smaller means better.
metric
A single number used to judge predictions, such as the share answered correctly.
noise
Random variation in data that carries no real signal, which a model can mistake for a pattern.
optimiser
The procedure that changes a model’s numbers to make its loss smaller.
overfitting
When a model memorises accidental details of its training examples and does worse on new ones.
parameters
The numbers inside a model that fitting adjusts, as opposed to the inputs it reads.
regularisation
Any penalty or limit that discourages a model from chasing noise in its training examples. The aim is better performance on new data.
supervised learning
A setting where each example carries the answer you want the model to predict.
test set
Examples kept out of fitting, used to estimate how the model will do on data it has not seen.
training set
The examples a model is fitted on; a score there is not an estimate of performance on new data.
weights
The numbers a model multiplies each input by before adding the results together.

Classical machine learning

Precision, recall and thresholds

A classifier outputs a score; a threshold turns it into a decision. Precision, recall and the confusion matrix tell you what that decision costs.

Part 7 of 10

Intermediate15 min readBack to Courses

The confusion matrix

A two-way decision has four outcomes. A true positive is a positive the model caught. A false positive is a negative it flagged. A false negative is a positive it missed. A true negative is a negative it correctly ignored. The four counts form the confusion matrix. Every two-way metric is a ratio built from them.

The two errors are not interchangeable. A false positive in a spam filter costs a missing email. A false negative costs an inbox full of spam. A false negative in a cancer screen can cost a life. A false positive costs a follow-up test. The matrix makes the trade visible before a single number hides it.

Precision and recall

Precision is the share of predicted positives that are real. It is true positives divided by true positives plus false positives. It answers a simple question: when the model says yes, how often is it right? Recall is the share of real positives that were found. It is true positives divided by true positives plus false negatives. It answers: of all the yes cases, how many did it catch?

F1 is the harmonic mean of the two. A harmonic mean is an average that leans toward the smaller value. F1 is high only when both numbers are high, so it is a common single-number summary. It also weights precision and recall equally. That is a choice, not a fact. If one error is much more expensive, F1 is the wrong summary. Use a cost-weighted measure, or fix a recall level and report the precision there.

The harmonic mean is sensitive to the smaller of the two. A model with precision 0.9 and recall 0.1 has an F1 near 0.18, not 0.5. That is intended. A model that finds almost nothing should not look half-good because its few predictions are right. The arithmetic mean would reward that trade. The harmonic mean refuses to.

The curve and the report

from sklearn.metrics import classification_report, precision_recall_curve

precision, recall, thresholds = precision_recall_curve(y_test, scores)
print(classification_report(y_test, predictions, digits=3))

Move the threshold

Run the sweep, then choose the threshold you would ship if a false positive cost five times as much as a false negative and explain the choice.

Output
Click "Run" to execute your code

Thresholds are a product decision

Most classifiers produce a score, and the threshold is applied afterwards. Lowering it catches more positives and admits more false positives. Raising it does the reverse. The model has not changed. Only the rule that turns its score into an action has. That makes the threshold a product decision, not a property of the model.

Prevalence is how common the positive class is. It changes what precision means. A test with 99 percent sensitivity and 99 percent specificity can still have low precision when the condition is rare. The false positives from the large negative group outnumber the true positives from the small positive group. Recall does not depend on prevalence in the same way. That is one reason to report the two numbers together.

A threshold sweep is the honest way to choose. Plot precision and recall against the threshold. Mark the points that meet the product constraint, and pick one with the cost in view. The curve also shows whether the choice is fragile. If a small change in threshold moves precision a lot, the deployed behaviour depends more on the score spread than on the model.

Accuracy on an imbalanced dataset is usually a vanity metric. A model that never predicts the rare class can score 95 percent and be useless for the task that motivated it.

Choosing a threshold

Start with the cost of each error. Then pick the point that minimises the expected cost at the prevalence you expect in production. If the costs are hard to state, fix a recall floor. Choose the highest precision that meets it, or do the reverse. Either way, report the whole curve. Do not report one point chosen after looking at the test set.

Calibration is a separate question. A model can rank well and still be overconfident. The rows it scores at 0.8 may be positive only 60 percent of the time. A threshold on a raw score does not need calibration. A decision that compares the chance to a cost does. Check the reliability curve before treating a score as a chance.

Calibration can be fixed after training with a monotone mapping. A monotone mapping never reorders the scores. Isotonic regression and a logistic fit on the scores are two examples. Such a mapping changes the probabilities without changing the ranking. So it changes thresholds but not the precision-recall curve. It is a useful repair when the score feeds a cost calculation. It is not a substitute for checking the ranking on held-out data.

The accuracy trap

Compute the confusion matrix and the four metrics, then compare the model with an always-negative baseline and decide which is more useful.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What does precision measure?