Colour and normalisation

Colour is a coordinate system

RGB, which stands for red, green and blue, is additive: three primaries combine to make a colour, and equal amounts make gray. It is convenient for displays and awkward for reasoning, because a change in lighting moves all three coordinates at once. HSV separates hue, saturation and value, so “reddish” and “bright” become different axes. Lab tries to make distance match perceived difference.

Most image models take RGB because that is what the pretrained weights saw. Converting to another space is a choice with consequences. It can make a task easier, and it always breaks compatibility with a pretrained stem. Do the conversion deliberately, document it, and apply the same conversion to training and inference.

Gamma is the trap inside RGB. It is a curve that maps stored values to light. Most images are stored with a transfer curve that is not linear in light, and most models treat the stored values as if they were. Converting to linear light before averaging or blending is more correct physically. It also changes the numbers a pretrained model expects, so it is a choice to make once and keep.

Normalisation is part of the model

A raw pixel is an integer from 0 to 255. A network wants a small float, roughly centred on zero, with a consistent scale across channels. The usual recipe divides by 255. It then subtracts a per-channel mean and divides by a per-channel standard deviation. The result has mean near zero and standard deviation near one on the training set.

The statistics are not universal constants. ImageNet means are a sensible default when the weights were pretrained on ImageNet. They are the wrong choice for microscopy or satellite imagery. Compute the mean and standard deviation of your own training set, store them with the model, and use exactly those numbers at inference.

Per-channel statistics are the default for colour images, because the channels have different distributions. A single mean and standard deviation over all channels is simpler and loses that information. For grayscale, or for a dataset where colour is irrelevant, one set of numbers is fine. The important part is that the same numbers are used everywhere.

To tensor, then normalise

from torchvision import transforms

train_transform = transforms.Compose([
    transforms.Resize(256),
    transforms.RandomCrop(224),
    transforms.ToTensor(),                       # 0..1, channels first
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225]),
])

image = train_transform(image)
print(image.shape, image.mean(), image.std())

Train statistics only

Computing the mean and standard deviation over the whole dataset before splitting leaks information from the test set into training. The leak is small, but it is real. It makes the reported score slightly optimistic in a way no loss curve reveals. Fit the statistics on the training split, and apply them unchanged everywhere else.

Batch normalisation is a different mechanism with the same name. It normalises each batch using that batch’s statistics during training and a running average during evaluation. A small batch makes those statistics noisy, which is why batch norm and small batches do not mix. Layer norm normalises across features instead. It is the usual choice when the batch is not the right unit.

Batch norm also changes what the model does at inference. The running statistics are an average over many batches, so a single example is normalised with a distribution it did not come from. That is deliberate, and it is why evaluation mode must be set. Training mode on a batch of one divides by a variance of nearly zero.

Colour augmentation and invariance

Colour jitter changes brightness, contrast, saturation and hue by small random amounts. It teaches the model that a red car in shade is still a red car, which is usually true. It can also destroy a diagnosis that depends on an exact stain colour. The strength is therefore a property of the task, not a default.

Grayscale conversion and channel swaps are stronger priors. Turning an image grayscale says colour carries no label information. If it does, the augmentation is a bug. The honest test is to apply the transform to a batch and look at it. If you cannot tell what the label should be, the model cannot either.

Order matters when several transforms are combined. Resize before crop gives a different scale than crop before resize. Colour jitter before normalisation is applied to the raw range; after it, to standardised values. Write the pipeline as an ordered list and read it aloud. Most augmentation bugs are a transform in the wrong position.