Pooling and stride

Why reduce the resolution

A feature map, the grid of responses produced by one layer, is expensive to keep at full resolution. Halving the height and width quarters the number of values, and every later layer inherits that saving. Reducing resolution also enlarges the receptive field. The receptive field is the part of the input that can affect one output value. A neuron deep in the network therefore sees a larger part of the image without a larger kernel.

Max pooling takes the largest value in each window and passes it on. It keeps the strongest response to a pattern and discards where exactly in the window it occurred. That gives a small amount of translation tolerance. Average pooling keeps the mean instead, which retains the overall level and blurs the peak. Neither has any parameters.

Translation tolerance is the useful side effect. A max pool over a 2 by 2 window reports the same value. The peak can sit at the top-left or the bottom-right of the window. A small shift in the input therefore does not change the output much. It is tolerance, not invariance. A shift larger than the window still moves the response.

Stride and the size formula

Stride is the distance the window moves between positions. A stride of one evaluates every position and produces an output almost as large as the input. A stride of two skips every other position and roughly halves the output. The same formula covers convolution and pooling. It is input minus kernel, plus twice the padding, divided by the stride, plus one.

With a 2 by 2 window and stride 2 on a 6 by 6 input, the output is 3 by 3. The windows do not overlap. With a 3 by 3 window and stride 2, the windows overlap and the output is 2 by 2. The last full window starts at position 2, and there is no room for another at 4. Integer division is why the edges sometimes get dropped.

Framework flags decide what happens at the boundary. PyTorch floors by default and has a ceil mode that keeps a partial window. That mode changes the output size by one. ONNX and some other runtimes default differently. When a model exported from one framework disagrees with another by a single row, this is the first place to look.

Pooling layers in PyTorch

import torch
from torch import nn

x = torch.randn(8, 16, 32, 32)

pool = nn.MaxPool2d(kernel_size=2, stride=2)
print(pool(x).shape)          # (8, 16, 16, 16)

average = nn.AdaptiveAvgPool2d(1)
print(average(x).shape)       # (8, 16, 1, 1)

Pooling against strided convolution

A max pool is a fixed rule. A strided convolution is a learned downsampling. The learned version can choose what to keep, at the cost of parameters and compute. Modern architectures often use strided convolutions in the stem, the first few layers. They reserve pooling for the classifier head, the last layers that produce class scores. Max pooling remains common, though, and is still the easiest way to explain the idea.

The classifier head is where global average pooling replaced the stack of dense layers. Global average pooling averages each channel over all positions and produces one number per channel. That is exactly what a linear classifier needs. It has no parameters, it accepts any input size, and it removes the enormous first dense layer that used to dominate the parameter count.

Strided convolution is not always better. It learns which pixels to keep, but it also learns a sampling pattern that can alias. Aliasing is a false pattern created when a signal is sampled too coarsely. A stride of two on a high-frequency texture can miss the very pattern the next layer needs. Pooling has the same problem. Both are reasons to blur slightly before downsampling in classical pipelines.

The receptive field

The receptive field is the region of the input that can affect one output value. A single 3 by 3 convolution sees 3 by 3. Two of them see 5 by 5, because the second kernel reads a neighbourhood that already contains the first. Three see 7 by 7. Depth grows the field linearly when the resolution is preserved.

Pooling and stride grow it faster. A 2 by 2 pool doubles the effective spacing. A stack of convolutions after it then covers twice the input for the same number of layers. That is the real reason downsampling is everywhere. It lets a deep network see the whole image without an impractical number of layers or an enormous kernel.

The theoretical field and the effective field differ. Not every input in the window contributes equally. The centre usually matters more, and a random weight pattern may ignore the edges entirely. Measure the effective field by perturbing pixels and watching the output. Or remember that the formula is an upper bound.