EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

accuracy
The share of predictions that are correct, counting every class the same.
additive
Built by summing separate contributions, so each part’s effect can be read on its own.
baseline
A reference score from a rule so plain it needs no learning, such as always naming the most common answer.
features
The input columns for one example — the numbers a model is allowed to read before it predicts.
fitting
Adjusting a model’s numbers so its predictions get closer to the answers in the training data.
generalisation
How well a model performs on examples it never saw while being fitted.
hypothesis class
The family of shapes a model is allowed to use, such as a straight line or a tree.
label
The correct answer for one example — what you want the model to predict.
loss
A single number measuring how wrong a prediction is; smaller means better.
metric
A single number used to judge predictions, such as the share answered correctly.
noise
Random variation in data that carries no real signal, which a model can mistake for a pattern.
optimiser
The procedure that changes a model’s numbers to make its loss smaller.
overfitting
When a model memorises accidental details of its training examples and does worse on new ones.
parameters
The numbers inside a model that fitting adjusts, as opposed to the inputs it reads.
regularisation
Any penalty or limit that discourages a model from chasing noise in its training examples. The aim is better performance on new data.
supervised learning
A setting where each example carries the answer you want the model to predict.
test set
Examples kept out of fitting, used to estimate how the model will do on data it has not seen.
training set
The examples a model is fitted on; a score there is not an estimate of performance on new data.
weights
The numbers a model multiplies each input by before adding the results together.

Classical machine learning

Logistic regression

A linear score passed through a sigmoid gives a probability. The loss is log loss, and the decision boundary is still a line.

Part 5 of 10

Intermediate16 min readBack to Courses

From score to probability

Logistic regression computes a linear score, then squashes it through a sigmoid. The sigmoid is an S-shaped curve. It maps any real number into the range zero to one. The output can be read as a chance between zero and one. A score of zero becomes 0.5. Large positive scores approach one. Large negative scores approach zero.

The score is a log-odds. Odds compare the chance of an event with the chance against it. If the chance is p, the odds are p divided by one minus p. The log of that ratio is the linear part. So a weight of 0.7 means the log-odds rise by 0.7 for a one-unit rise in the feature. That multiplies the odds by about two. The intercept sets the baseline log-odds.

Log loss

The loss is the negative log likelihood of the observed labels. For a positive example it is minus the log of the predicted probability. For a negative one it is minus the log of one minus that probability. A confident correct prediction costs almost nothing. A confident wrong one costs a lot. That is the behaviour a probability model should have.

There is no closed form for the weights. The loss is convex, so gradient descent converges to the global optimum. The gradient has the same shape as in linear regression. It is the error times the feature, averaged over the rows. The difference is that the prediction has passed through the sigmoid. The error is therefore bounded between minus one and one.

Convexity is why the fit is repeatable. A convex loss has one bottom. Gradient descent from any reasonable start reaches the same weights, up to rounding error. That is not true for a neural net. There the same data and different seeds can land in different basins. Logistic regression is a good first classifier partly because its training is boring.

Probabilities from scikit-learn

from sklearn.linear_model import LogisticRegression

model = LogisticRegression().fit(X_train, y_train)
print(model.coef_, model.intercept_)

probabilities = model.predict_proba(X_test)[:, 1]
print(probabilities[:5])

Fit a probability curve

Run the gradient steps, then raise the threshold to 0.7 and count how many predictions change without retraining the model.

Output
Click "Run" to execute your code

Coefficients are log-odds

A positive weight means the feature raises the chance. Its size is the change in log-odds per unit. The effect on the chance is not constant. Near 0.5 the curve is steep. Near zero or one it flattens. So comparing weights across features with different scales needs care. Scaling the features makes the sizes more comparable.

Correlated features share their effect, as in linear regression. Regularisation, a penalty on large weights, is common and often on by default. An L2 penalty shrinks the weights toward zero. That stabilises the fit when columns are collinear. The penalty is a choice, not a neutral default. Tune it on validation data, not at whatever the library picked.

Regularisation also changes what the weights mean. An L2 penalty shrinks them toward zero. A weight is then no longer the pure best fit to the data. It is the effect under a budget. Comparing a penalised model with an unpenalised one by weight size is therefore wrong. Compare predictions, or refit without the penalty when you want to read the weights.

Report the probability, not only the label. A model that says 0.51 and one that says 0.99 make the same prediction at a 0.5 threshold, but they are not equally useful to a decision that has a cost.

The boundary is linear

The set of points where the score is zero is a line, a plane or a hyperplane. Everything on one side is predicted positive. Everything on the other is predicted negative, before any threshold other than 0.5. A linear boundary cannot represent an interaction, a joint effect of two features. An example is “positive when both features are high or both are low”. That is the XOR pattern.

The fix is to add features. Products, squares, splines or a learned encoding all widen the input. A logistic regression on built features is still a linear model in its weights. Its boundary in the original space can curve. A neural net does the same thing by learning the features. So the line between linear and nonlinear is about the boundary, not the name of the model.

Class weights are the other common fix. When positives are rare, the loss can weight a false negative more heavily. That shifts the boundary toward catching more positives. It changes the fitted model, not only the threshold. Check it on the same held-out set as any other choice. The goal is a rule that reflects the cost, not a model that looks balanced in the report.

Probabilities, odds and logits

Print the table, then add a weight of two to a feature and shift every logit by twice the feature value.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What does the sigmoid do to a linear score?