EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

anchor
A reference box of a fixed size and shape that a detector adjusts to fit an object.
augmentation
Random changes to training images that keep the correct answer intact, such as crops, flips and brightness shifts.
bounding box
Four numbers marking an object’s edges, either as two corners or as a centre, width and height.
channel
One grid of values per colour or per learned pattern; an RGB image has three such grids.
convolution
Sliding a small grid of weights across an image, multiplying and summing at each position.
feature map
The grid of responses one kernel produces when slid across an image.
IoU
The overlap area of two boxes divided by the area they cover together; one means identical, zero means no overlap.
kernel
The small grid of weights a convolution slides over an image.
normalisation
Rescaling values to a common range or spread, often using statistics from the training images.
padding
Extra values, usually zeros, added around an image’s border so a sliding window can reach the edge.
pixel
One number in an image grid, giving brightness or colour strength at a single position.
pooling
Summarising each small window of a feature map into one value, such as its maximum or average.
receptive field
The part of the input that can affect one output value.
stride
How far a sliding window moves between positions; larger steps shrink the output.

Computer vision from the ground up

Augmentation

Augmentation is a prior written as transforms: it tells the model which changes do not matter. The label has to survive every transform, or the model learns noise.

Part 5 of 10

Intermediate14 min readBack to Courses

What augmentation buys

A model that has seen a cat only from the left learns the left side of a cat. A horizontal flip gives it the other side for free, and the label is unchanged. Flips, crops, small rotations and colour jitter all encode an invariance the task already has. An invariance is a change that leaves the label the same. These transforms multiply the effective dataset without another label.

It is also a regulariser. Every augmented view is a slightly different input with the same target, so the model cannot memorise one exact pixel pattern. Augmentation therefore often helps more than dropout on image tasks. It adds variation where the data actually lives, rather than noise in the activations.

The gain has a ceiling. Augmentation expands the neighbourhood of each example, but it cannot create a new class or a new scene. Once the model has seen the same image flipped, cropped and jittered a hundred times, another pass adds little. The curve of validation accuracy against augmentation strength usually rises, then flattens or falls.

The label has to survive

A horizontal flip turns a 6 into something closer to a 9. It turns text into mirrored text and a left-hand sign into a right-hand sign. If the label changes, the augmentation is teaching the model a contradiction. The rule is short: apply the transform, look at the result, and ask whether you could still label it with confidence.

Rotation has the same trap. A satellite image of a boat is fine upside down; a photo of a person is not. Cropping is fine for a large object and dangerous for a small one, because the crop can remove the only evidence. The strength of an augmentation is a claim about the data. The claim has to be true.

A useful test is to label a batch yourself after augmentation. If you hesitate on more than a few examples, the pipeline is too aggressive. The same test catches a crop that removes the object and a rotation that makes the scene implausible. Both are invisible in the loss until the model has already learned the wrong invariance.

Train and validation transforms

from torchvision import transforms

train_tf = transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ColorJitter(0.2, 0.2, 0.2),
    transforms.ToTensor(),
])

val_tf = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
])

# The validation set gets no random transform.
print(train_tf, val_tf)

Flip, rotate, crop

Apply the three transforms to the same grid and print each result. Then rotate twice and check that four rotations return the original orientation.

Output
Click "Run" to execute your code

Train only, and reproducibly

Augmentation belongs to the training pipeline. Validation and test get a deterministic resize, centre crop and normalisation. The number you report has to be comparable across runs. Applying random crops at test time gives a different score every evaluation and hides real changes in the model.

Random transforms need a seed if a run is to be reproducible. A DataLoader with workers needs that seed passed to each worker. Otherwise the sequence depends on process scheduling. The seed is not a detail. Without it, two runs that differ only in augmentation cannot be compared, and a regression cannot be reproduced.

Worker processes need their own seeds, and the order in which they finish is not guaranteed. PyTorch exposes a worker init function and a generator per worker for exactly this reason. If the augmentation is random and the run is not reproducible, the first suspect is the worker seed, not the model.

Look at a batch of augmented images before training. A pipeline that crops the object out of the frame is worse than no pipeline, and the loss curve will not tell you.

Augmentation is not a substitute for data

Augmentation can only express invariances you already know. It cannot invent a class, a viewpoint or a lighting condition that is absent from the data. It cannot fix a labelling rule that is inconsistent. When the test distribution differs in a way no transform covers, more data or a different data collection is the answer.

Mixup and cutmix are a different kind of augmentation: they combine two images and two labels. The model sees a blend and is asked to predict a blend of the targets, which smooths the decision boundary. They work well on classification benchmarks. They need care when the label is a box or a mask, because blending two objects creates a scene that never existed.

The honest summary is that augmentation is a hypothesis about invariance. State the hypothesis, test it on a batch, and measure it on validation. An augmentation that helps on one dataset and hurts on another is not a bug in the transform. It is evidence that the two datasets have different symmetries.

Random crops and coverage

Draw two hundred random 3 by 3 crops from a 6 by 6 image and count how often each pixel is used. The centre is covered more than the corners, which is a property of the sampling, not a bug.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

Which split gets random augmentation?