Train, validation and test

Three questions, three sets

The training set fits the parameters. The validation set chooses between models, thresholds and hyperparameters, the settings you pick before training. The test set estimates how the chosen model will do on data it has never seen. Each set answers a different question. Use one set for two jobs and the answer becomes optimistic.

The test set is a promise. You look at it once, after the decisions are made. Every peek turns it into a validation set. The final number then reports how well you tuned, not how well the model generalises. If you need to compare many candidates, use cross-validation on the training data. Keep the test set untouched.

The proportions matter as much as the names. A test set that is one percent of the data gives a score with a wide interval. A validation set that is one percent gives a noisy choice. A common split is 60/20/20 for small data. Use 98/1/1 when the dataset is large enough that one percent is still thousands of rows. The right ratio gives each question enough evidence.

Leakage is the default failure

Leakage is any information from the test set that reaches the model during training or selection. The obvious form is a duplicated row. The quiet forms are more common. One is scaling with the mean of the full dataset. Another is imputing with a median computed before the split. A third is choosing features by looking at all labels. A fourth is letting a time-series split train on the future.

The fix is to make preprocessing part of the model and fit it only on training data. A pipeline that scales, imputes and classifies is one object. Calling fit on the training split fits every step there. That structure makes the leak hard to introduce. A habit is less reliable than a structure.

A split and a pipeline

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=0
)
model = make_pipeline(StandardScaler(), LogisticRegression())
model.fit(X_train, y_train)
print(model.score(X_test, y_test))

Cross-validation when data is small

A single validation split is noisy when the dataset is small. The score depends on which rows happened to land there. K-fold cross-validation rotates the validation block through the data. It fits the model k times and reports the mean and spread. The spread matters as much as the mean. A model with a good average and a huge variance is a gamble.

The folds must respect the structure of the data. Group by customer, so the same customer does not appear in train and validation. Split by time, so the validation rows come after the training rows. Stratify by class, so a rare label is not missing from a fold. A random fold is the default. It is not the correct choice for every dataset.

Cross-validation is not free. It multiplies training time by the number of folds. It is awkward when the model is expensive or the data is streamed. It also does not create information. If the dataset has no examples of a rare subgroup, every fold will miss it equally. The estimate is honest about the sample. It says nothing about a population the sample failed to represent.

What a split cannot fix

A held-out set estimates performance on the same distribution as the data you split. Distribution here means the mix of cases the data came from. If the world shifts after collection, the estimate describes the past. A model trained on last year’s customers can be accurate on last year’s holdout. It can still be useless in the current traffic mix. Monitoring is the only way to see that happen.

Small test sets also carry wide uncertainty. Four correct predictions out of five is compatible with a true accuracy anywhere from roughly forty to ninety percent. Report the sample size with the score. Prefer an interval to a bare decimal. A number without its denominator invites a confidence it has not earned.

The split also has to match the deployment unit. That is the thing a prediction is made about. If predictions are made per customer but rows are per order, a random row split puts the same customer on both sides. That overstates performance. Grouping by customer, session or device is part of the evaluation design, not a preprocessing detail. The question is always what “unseen” means for the thing that will use the model.