EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

accuracy
The share of predictions that are correct, counting every class the same.
additive
Built by summing separate contributions, so each part’s effect can be read on its own.
baseline
A reference score from a rule so plain it needs no learning, such as always naming the most common answer.
features
The input columns for one example — the numbers a model is allowed to read before it predicts.
fitting
Adjusting a model’s numbers so its predictions get closer to the answers in the training data.
generalisation
How well a model performs on examples it never saw while being fitted.
hypothesis class
The family of shapes a model is allowed to use, such as a straight line or a tree.
label
The correct answer for one example — what you want the model to predict.
loss
A single number measuring how wrong a prediction is; smaller means better.
metric
A single number used to judge predictions, such as the share answered correctly.
noise
Random variation in data that carries no real signal, which a model can mistake for a pattern.
optimiser
The procedure that changes a model’s numbers to make its loss smaller.
overfitting
When a model memorises accidental details of its training examples and does worse on new ones.
parameters
The numbers inside a model that fitting adjusts, as opposed to the inputs it reads.
regularisation
Any penalty or limit that discourages a model from chasing noise in its training examples. The aim is better performance on new data.
supervised learning
A setting where each example carries the answer you want the model to predict.
test set
Examples kept out of fitting, used to estimate how the model will do on data it has not seen.
training set
The examples a model is fitted on; a score there is not an estimate of performance on new data.
weights
The numbers a model multiplies each input by before adding the results together.

Classical machine learning

Feature engineering

A model can only use what its columns expose. Scaling, encoding, missing values and ratios decide whether the algorithm ever sees the pattern you meant.

Part 8 of 10

Intermediate15 min readBack to Courses

The model sees columns

A feature is a number the model can compare, split or multiply. A raw table arrives with text, dates, missing values and units. Feature work turns those into numbers without destroying the meaning they carry. The choice of model is often the smaller half of the job.

The columns set the ceiling. A linear model cannot use a relation it cannot express. A tree can approximate an interaction, a joint effect of two columns, only by spending depth and data on it. A ratio or a difference that states the relation outright is worth more than another round of tuning.

Every column also carries a choice. Is a missing income zero, the median, or its own group? Is a date a timestamp, a weekday, or days since the last purchase? Those choices are part of the model. Write them down once, rather than making them differently in training and serving.

Scale changes the distance

Some models measure distance or penalise weights in whatever units they are given. Nearest neighbours, k-means, support vector machines and penalised linear models all do. An income in dollars and an age in years are not comparable. The income column will dominate every distance and every penalty unless it is scaled.

To standardise, subtract the mean and divide by the standard deviation. One unit then means one standard deviation. Min-max maps the column into a fixed range. That is useful for bounded inputs and for neural nets. Fit the scaler on the training split only. Then apply the same numbers to validation and test, or the test spread leaks into training.

Trees do not care about monotone transforms. A monotone transform keeps the order of the values. Scaling therefore does not change the order of a tree’s splits. It still changes penalised linear models, neural nets and anything that measures a distance. When the model is not a tree, scale first. It rarely hurts, and it often decides whether the model learns at all.

Preprocessing inside the pipeline

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

preprocess = ColumnTransformer([
    ("numeric", StandardScaler(), ["age", "income"]),
    ("categorical", OneHotEncoder(handle_unknown="ignore"), ["region"]),
])
pipeline = Pipeline([("preprocess", preprocess), ("model", model)])
pipeline.fit(X_train, y_train)
print(pipeline.predict(X_test))

The nearest neighbour changes

Run the query with raw and standardised columns, then add another person and see which neighbour wins before scaling.

Output
Click "Run" to execute your code

Categoricals and missing values

A nominal column, one whose values are names rather than ranks, has no order. One-hot encoding gives each value its own column. An ordinal column does have an order, and integer codes preserve it. The common mistakes are the two inversions. One-hot encoding an ordinal throws the order away. Integer-encoding a nominal invents an order the model will use.

Columns with many different values are harder. One-hot encoding a postcode creates thousands of sparse columns. Target encoding replaces the value with the mean label. It leaks unless it is cross-fitted, meaning fitted on other folds. Hashing keeps a fixed width at the cost of collisions. None of these is free. The choice should follow how many rows each value has.

A missing value is often a signal. A missing income can mean refused, unknown or not applicable. A separate flag column lets the model use that difference. Imputing with the median, which means filling in the missing value, is a reasonable default. It is not a neutral act. Check whether the missing rows behave differently before you choose it.

Fitting a scaler, an imputer or a target encoder on the full dataset leaks the test distribution into training. Put every fitted transform inside the pipeline so cross-validation refits it on each fold.

Interactions, time and selection

A ratio or a product can encode a relation that no single column contains. Examples are price per room, distance over speed, and signal to noise. If the target is plausibly a ratio, give the model the ratio. Do not ask it to discover division from two additive columns.

Dates become features by splitting them into parts: hour, weekday, month, days since an event. A cyclical feature such as hour should be encoded as sine and cosine. Then 23:00 and 00:00 are neighbours instead of opposite ends of a line. A raw hour number tells the model that midnight is far from one in the morning.

More columns are not more information. Constants, duplicate columns and anything built from the label add variance or leakage. Select with a held-out score, not a training-set correlation. Keep the selection inside the cross-validation loop. Then the number you report is not the number you selected on. A feature built one way in training and another way in serving is the same kind of bug. It stays invisible until the model meets live data.

A ratio the model cannot build

Fit a line on distance alone and on distance divided by speed, then add a fixed two-minute stop and refit both models.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

Why fit a scaler on the training split only?