Learning from examples

Three choices, one dataset

Every supervised model makes three choices. A supervised model is one trained on labelled examples. The hypothesis class is the family of shapes the model is allowed to use: a line, a tree, a small network. The loss is a number saying how wrong a prediction is. The optimiser is the procedure that adjusts the weights to make that number smaller. Change any one of the three and you have a different learner, even when the data is identical.

The data supplies examples. Each example is a row of features, the input columns, and a label, the answer you want to predict. Learning means finding the parameters in the chosen class that make the loss small on those examples. That process is fitting, and fitting is not the same as understanding. A model can match every training row and still be wrong on the next one. So the next chapter is about evaluation rather than optimisation.

The loss is not a detail you can pick at random. Squared error treats a miss of ten as a hundred times worse than a miss of one. Absolute error treats the same miss as ten times worse. A loss for sorting rows into classes can also weight one class more than another. To choose a loss is to choose which mistake the model should avoid. That is a decision about the problem, before it is a decision about the maths.

Features are the interface

A model never sees a customer, a sentence or an image. It sees numbers. Feature engineering is the work of turning a raw case into a row of numbers. It decides what the model can learn at all. A useful feature makes the pattern easier to see. A missing one sets a ceiling on accuracy, and no amount of tuning will lift it.

The label is part of that interface too. A label that two annotators would assign differently is noise. A label that leaks information from the future is worse than noise. Before you choose a model, look at one row. Ask whether a careful person could make the prediction from those columns alone.

The same three choices in scikit-learn

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0
)
model = LogisticRegression().fit(X_train, y_train)
print(model.score(X_test, y_test))

Start with a baseline

A model that always predicts the most common label is a real model. It is surprisingly hard to beat when one class dominates the data. It gives you a number to beat and a check on your metric. If the baseline scores 0.9 accuracy, a model at 0.91 has learned very little. Report the baseline next to every later score.

A second baseline is the simplest rule a domain expert would try. That might be a threshold, a keyword match or last year’s value. If a one-line rule matches the learned model, the extra complexity has not paid for itself. A baseline does not have to be good. Its job is to make the improvement measurable.

A baseline can also be a previous production model. The new model may beat it by a small margin on a noisy test set. Then the honest conclusion is that the two are hard to tell apart at the current sample size. The comparison needs the same split and the same metric. It also needs a sense of how much the metric moves from one sample to the next.

What learning means

Training minimises empirical risk. That is the average loss on the examples you have. Generalisation is the average loss on examples you have not seen. The gap between the two is what regularisation tries to control. A penalty on complexity is one such technique. It also explains how a model that is too flexible can lose to one that is too simple.

Those assumptions are the inductive bias. A line assumes the relationship is straight. A tree assumes the boundary follows the axes. A neural network assumes little, so it needs more data to compensate. When a model has more flexibility than the data can support, it fits the noise. That failure is overfitting. No class is best for every problem. So the first step is to look at the data, not to reach for the newest model.

The no-free-lunch idea is often stated too strongly. It says that no class dominates over every possible problem. It does not say that all classes are equally useful for yours. Domain knowledge narrows the space. A linear model is a good bet when the relationship is roughly additive. A tree is a good bet when the boundary is a set of thresholds. The data then tells you whether the bet paid off.