Classification pipelines

The pipeline in order

An image classifier is a sequence of decisions before it is a network. Split the data, fit the transforms on the training split, build the loader, choose the loss, run the loop, and evaluate on the held-out set. That set is the data kept out of training. Each step has a failure mode that looks like a model problem from the outside. A bad split looks like overfitting, and a bad normalisation looks like a bad architecture.

The test set is touched once, at the end. Every choice made by looking at the test score is a choice that has been tuned on the test set. The final number then stops being an estimate of generalisation, which is performance on new data. Use validation for choices, and keep test for the report. If the report matters, that discipline is the report.

The class mapping is part of the pipeline. ImageFolder sorts directory names alphabetically and assigns indices in that order. A model trained with one mapping and served with another returns confident, wrong labels. Save the mapping with the weights, and assert it at load time, the same way you assert an input shape.

From logits to a decision

The model returns one logit per class. A logit is a raw score, before any normalisation. Softmax turns the logits into a distribution, and argmax picks the largest. The predicted class is the same with or without the softmax, because softmax is monotonic. A monotonic function keeps the order of its inputs. The probability is still useful for a threshold, a ranking, or a human reading the output. A confidence of 0.6 is not a promise that the model is right 60 percent of the time, unless it is calibrated.

Top-k accuracy asks whether the true class is among the k highest scores. It is the right metric when a human or a later stage will consider several candidates. It also hides a model that ranks the right class second every time. Report top-1 as well, and report per-class numbers when the classes are not equally important.

Calibration is the gap between confidence and accuracy. A model that says 0.9 and is right 70 percent of the time is overconfident. That matters when the score drives a decision. Reliability diagrams and expected calibration error measure it. Temperature scaling is the cheap fix, fitted on validation and applied at inference.

A dataset and a loader

from torchvision import datasets, transforms
from torch.utils.data import DataLoader

train_tf = transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
])
val_tf = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
])

train_set = datasets.ImageFolder("data/train", transform=train_tf)
val_set = datasets.ImageFolder("data/val", transform=val_tf)

train_loader = DataLoader(train_set, batch_size=32, shuffle=True)
val_loader = DataLoader(val_set, batch_size=64, shuffle=False)

print(train_set.classes)
print(len(train_set), len(val_set))

Metrics that survive imbalance

Accuracy is the fraction of predictions that are correct, and it is dominated by the largest class. A detector for a condition that occurs in five percent of cases can be 95 percent accurate by always saying no. The confusion matrix shows where the errors are. It is a table of true class against predicted class. Per-class recall shows which classes are being missed, and macro averaging gives every class the same weight.

Choose the metric before training, from the cost of the two mistakes. A false negative in a screening test is not the same as a false positive. If the cost is asymmetric, a threshold is part of the model. The operating point should be chosen on validation data, not left at 0.5 by default.

The confusion matrix is the source of all of it. Rows are true classes and columns are predictions, or the reverse. Pick one convention and write it down. A matrix that looks transposed is a common review comment, and it changes every per-class number that follows. Print the labels beside the rows and columns, not only the counts.

Leakage and the split

Randomly splitting rows is wrong when rows are not independent. Frames from the same video, photos of the same patient and near-duplicate images from a scrape all put the same object on both sides. The model then recognises the object rather than the class. The validation score is optimistic in a way that only the test set discovers.

Split by group, by subject or by time, depending on what the deployment sees. Fit every preprocessing statistic on train only. Apply augmentation after the split, never before. These rules are boring. They are the difference between a number that predicts production and a number that predicts the notebook.

Temporal splits are the right choice when the deployment predicts the future. A random split lets the model see next week’s data while learning last week’s, and the validation score is a leak. Group splits are right when the unit is a person, a device or a location. The split should mirror the unit that will be new at inference.