Overfitting and regularisation

The gap is the signal

Training loss measures how well the model fits the examples it has seen. Validation loss measures how well it fits examples it has not. Capacity is how flexible the model is. With enough capacity, the first loss can go to zero while the second rises. The distance between them is the clearest sign that the model has started to memorise rather than generalise. To generalise is to do well on examples it has never seen.

More data is the best regulariser, because it makes memorisation harder without constraining the model. Regularisation is any constraint that makes the easy answer harder. When more data is not available, you have four options. Reduce capacity, constrain the weights, add noise to the inputs or the activations, or stop before the validation loss turns. Each is a different way of making the easy answer harder.

The classical framing is bias and variance. Bias is error from a model that is too simple. Variance is error from a model that is too sensitive to its training set. A model with high bias underfits both sets. One with high variance fits the training set and not the validation set. Capacity moves a model along that axis, and regularisation is how you move it back without collecting more data. The trade is never eliminated. It is only placed where the task can tolerate it.

Weight decay and dropout

Weight decay adds a penalty proportional to the size of the weights. The optimiser then has to trade fit against magnitude. Large weights let a model carve sharp boundaries around individual points. Shrinking them smooths the function. In AdamW the decay is decoupled from the adaptive step, so the strength means what it says.

Dropout zeroes each unit with probability p during training. A unit is one neuron in a layer. It then rescales the survivors by one over one minus p, so the expected activation is unchanged. At evaluation time nothing is dropped and no rescaling happens. The usual reading is that dropout prevents units from co-adapting. The practical reading is that it injects noise into every layer, which is a strong regulariser when data is scarce.

Dropout is applied per example, not per batch, and the mask changes every forward pass. At inference the full network is used, which is why the rescaling has to happen during training. If you forget the rescale, the expected activation at test time is smaller by the drop probability. Every downstream layer then sees a different distribution.

Dropout and decay in a training step

import torch
from torch import nn

model = nn.Sequential(
    nn.Linear(64, 128),
    nn.ReLU(),
    nn.Dropout(p=0.3),
    nn.Linear(128, 10),
)
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)

model.train()
logits = model(torch.randn(32, 64))
loss = nn.functional.cross_entropy(logits, torch.randint(0, 10, (32,)))
loss.backward()
optimizer.step()

model.eval()   # dropout is now the identity
with torch.no_grad():
    print(model(torch.randn(32, 64)).shape)

Early stopping and augmentation

Early stopping trains until the validation loss has not improved for a set number of epochs. It then restores the weights from the best epoch. It is cheap, and it needs no change to the model. It works because the best generalising point is usually before the training loss reaches zero. Keeping the last epoch instead of the best one is a common and quiet way to ship a worse model.

Augmentation is a prior expressed as transformations: flips, crops, rotations, colour changes. Augmentation means random changes applied to training images. An invariance is a change that leaves the label the same. Augmentation teaches the invariances the task actually has, and it multiplies the effective dataset without new labels. It also changes the distribution. An augmentation that destroys the label, such as a horizontal flip of a digit, teaches the model something false.

Patience is the parameter that makes early stopping honest. Too small and the run stops on a noisy dip. Too large and it wastes compute and may still end past the best point. A patience of five to twenty epochs is typical. The validation loss should be measured on enough examples that a single batch cannot move it.

Regularisation is a trade

Every constraint costs some training fit. A little weight decay smooths the boundary and helps. Too much forces every weight toward zero, and the model underfits. Dropout at 0.5 is standard for a wide classifier and disastrous for a small one. The right strength is a property of the dataset and the capacity. It is not a constant to copy.

The interactions matter too. Dropout and batch normalisation behave badly together in some orders. Batch normalisation rescales each batch to a stable range, and dropout changes the variance it sees. Weight decay on biases and normalisation parameters is usually a mistake. No regulariser can fix a leaking validation split, either. If the same subject or near-duplicate frame appears in train and test, the number is optimistic no matter what the loss curve says.

The discipline is to decide the regularisation budget before looking at the test score. Try a small grid on validation. Pick the simplest setting within noise of the best, and report that. Choosing the single best point from a large grid on a small validation set is another way to overfit. The validation set is the victim that time.