Capstone: a convolution layer and its output

The layer, completely

The input has C_in channels, each of height H and width W. The layer owns C_out filters. A filter is the full set of weights that produces one output channel. Each filter is a stack of C_in kernels of size K by K. There is one bias per output channel, and the bias is a learned offset. The output has C_out channels. Every output value is the sum of a K by K by C_in patch times the corresponding filter, plus the bias.

The parameter count follows directly: C_out times C_in times K times K, plus C_out. A layer with 3 input channels, 32 output channels and a 3 by 3 kernel has 896 parameters. That is small compared with a dense layer on the same input. It is the reason convolutional networks can be deep without exploding.

Padding and stride are arguments of the layer, not separate operations. Padding adds zeros around the input before the sliding starts. Stride changes how far the window moves. Both change the output size, and both are recorded in the layer configuration. A saved model can therefore reproduce its own shapes without the original code.

The forward pass as loops

The definition is six nested loops: output channel, output row, output column, input channel, kernel row, kernel column. The innermost line multiplies an input value by a weight and adds it to the accumulator. The accumulator is the running total. Every optimised implementation computes exactly this, in a different order. The loops are flattened into matrix multiplies.

Writing the loops once is the best way to understand the shapes. The input channel loop is the one people forget. A filter is not a single 3 by 3 grid when the input has depth. It is a 3 by 3 grid per input channel, and all of them contribute to one output value.

The loop order is arbitrary. Putting the output channel outermost makes the weight reuse obvious. Putting the kernel loops innermost makes the memory access contiguous. Real implementations use im2col or a direct kernel with register blocking, but they compute the same sum. im2col rearranges image patches into columns so the convolution becomes one matrix multiply. The loops are the specification, not the implementation.

The layer and its parameter count

import torch
from torch import nn

layer = nn.Conv2d(3, 32, kernel_size=3, padding=1, bias=True)
x = torch.randn(1, 3, 64, 64)

print(layer(x).shape)     # (1, 32, 64, 64)
print(layer.weight.shape) # (32, 3, 3, 3)
print(layer.bias.shape)   # (32,)

What the output means

Each output channel is a feature map: a grid of responses to one learned pattern. A high value means the pattern was present at that position. The first layer of a trained network usually learns edges and colour blobs. Deeper layers learn textures and parts. None of that is imposed by the architecture, only by the data and the loss.

A channel that is zero everywhere is dead. Its filter never produced a positive response, or the ReLU after it never opened. That can be a sign of a learning rate that was too high. It can also mean an initialisation that was too aggressive, or a filter that is irrelevant to the task. Visualising a few channels is a cheap check before a long run.

The bias shifts the threshold of a feature map. A large negative bias means the channel only fires for a very strong response. A large positive one means it fires almost everywhere. After batch norm the bias is often redundant, which is why layers that use it are sometimes created with bias=False.

From layer to network

A convolutional network stacks these layers, inserts a nonlinearity after each one, and downsamples between stages. The early stages keep the resolution and learn local patterns. The later stages trade resolution for channels and learn combinations of patterns. A global average pool collapses the last feature map to one vector per channel, and a linear layer turns that into logits.

The receptive field, the parameter count and the output shape are the three numbers to track. If a stack loses too much resolution, small objects disappear. If the channel count grows too fast, the parameters and the memory follow. If the output shape is wrong, the error is usually one missing padding or one stride applied twice.

The next step is to train the same stack on a real dataset. CIFAR-10 is small enough to run on a laptop and varied enough to need augmentation, normalisation and a schedule. Everything in this course appears there: shapes, kernels, pooling, colour, augmentation, the loop and the metrics. The layer you built by hand is the one the framework runs.