Convolution by hand

The sliding window

A convolution takes a small grid of weights, called the kernel, and places it over the top-left corner of the image. It multiplies each kernel value by the pixel underneath, sums the products, and writes one number to the output. Then it moves the kernel one pixel to the right and repeats. When the row ends, it moves down.

With a 3 by 3 kernel and a 6 by 6 image, the kernel fits in four positions across and four down. The output is 4 by 4. The general formula is the input size minus the kernel size, plus twice the padding, divided by the stride, plus one. Padding is zeros added around the border, and stride is how far the window moves each step. Every shape in a convolutional network comes from that one line.

A concrete case makes it stick. A 32 by 32 input, a 3 by 3 kernel, padding 1 and stride 1 give 32 by 32. The padding replaces the two rows and columns the kernel would have eaten. Remove the padding and the answer is 30. Keep the padding and double the stride, and it is 16.

Kernels as feature detectors

The kernel is a pattern matcher. A kernel with a bright centre and negative neighbours responds to a dot or a thin line. A Sobel kernel responds to a change in brightness along one axis, which is an edge. A box of ones divided by nine averages the neighbourhood and blurs. The operation is the same; the weights differ.

A convolutional layer learns several kernels at once, and each one produces one output channel. A channel here is one feature map in the output. The weights are shared across every position, so a vertical-edge detector is the same detector at the top and the bottom of the image. That sharing is why a convolutional layer has far fewer parameters than a dense layer on the same input. It is also why the layer generalises across translation, which is a shift in position.

A trained network builds a hierarchy. The first layers respond to edges and colour transitions. The middle layers combine them into corners, textures and parts, and the last layers respond to objects. Nobody assigns those roles. They emerge because the loss rewards whatever features make the classification easier.

A convolution layer in PyTorch

import torch
from torch import nn

layer = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, padding=1)
x = torch.randn(8, 3, 32, 32)

y = layer(x)
print(y.shape)                              # (8, 16, 32, 32)
print(layer.weight.shape)                   # (16, 3, 3, 3)
print(sum(p.numel() for p in layer.parameters()))   # 448

Padding, channels and parameters

Without padding, every convolution shrinks the image by the kernel size minus one. A stack of five 3 by 3 layers loses ten pixels in total, five on each side. That matters when the image is small. Zero padding keeps the size, at the cost of a dark border that the kernel can mistake for an edge. The choice is a design decision, not a detail.

The parameter count is the number of learned weights and biases. A bias is a learned offset added to every output of a channel. A layer with three input channels and sixteen output channels has sixteen kernels. Each kernel has three planes, one per input channel. Every output value sums over all three input planes. The parameter count is sixteen times three times three times three, plus sixteen biases: 448. If your printout says 432, you forgot the bias.

A 1 by 1 convolution is a special case worth knowing. Its kernel is a single weight per input-output channel pair, so it mixes channels at every position without looking at neighbours. It is used to change the channel count cheaply. It is exactly a dense layer applied to each pixel with the same weights.

Convolution is not magic

The operation encodes two priors. A prior is an assumption built into the model before it sees data. The first is locality, because each output looks only at a small neighbourhood. The second is translation equivariance, because the same kernel is applied everywhere. Translation equivariance means a shift in the input moves the response with it. Those priors are right for images and wrong for tabular data, where neighbouring columns have no spatial relation. A dense layer has no such assumption and needs more data to learn one.

It is also not invariant to rotation, scale or mirroring. A vertical-edge detector does not fire on a horizontal edge. A model trained on upright objects may fail on rotated ones, unless augmentation teaches it otherwise. The architecture supplies the prior; the data has to supply the rest.

The cost is real. A layer with 256 input channels, 256 output channels and a 3 by 3 kernel has 590,080 parameters. It performs about 590,000 multiply-adds per pixel. At 56 by 56 that is nearly two billion operations for one layer. Channel counts and resolutions are therefore balanced rather than maximised.