Backpropagation by hand

The chain rule on a graph

Every operation in a forward pass has a local derivative. That is how much the operation’s output changes for a small change in one input. Addition passes the gradient through unchanged, multiplication scales it by the other input, and ReLU passes it or blocks it. Backpropagation multiplies the gradient arriving from above by each local derivative. It then sends the result to the inputs.

When an output feeds two operations, both paths contribute, and the contributions add. That is the only subtle rule. In the expression z = x times y plus y, the derivative with respect to y is x plus one. The reason is that y reaches z twice: once through the product and once directly.

A concrete number helps. If x is 2 and y is 3, a small change in y changes the product by 2 and the direct term by 1. The total derivative is therefore 3. Backpropagation finds that 3 by adding the two paths, not by differentiating the whole expression symbolically. Every framework is doing this arithmetic, one node at a time.

Reverse topological order

A node can only send a complete gradient once every consumer has sent its own. The safe order is the reverse of a topological sort. A topological sort puts every node after the nodes it depends on. Compute the forward values, record the graph, then walk the nodes in reverse. Each node then comes after all of its consumers. A naive recursion that revisits shared nodes either double-counts or misses a path.

This is exactly what an autograd engine does. Autograd is the machinery that records operations and computes their gradients. The forward pass builds a graph of operations. The backward pass seeds the output gradient with one and calls each node in reverse order. Frameworks hide the graph, but the order is not an implementation detail. It is the reason shared weights and branches get the right sum.

Graphs are freed after backward by default. Calling backward twice on the same graph without retain_graph=True raises an error. The intermediate buffers have already been released. That is a feature. Keeping every activation alive for a second pass would double the memory for a case that is rare. If you need two losses, sum them before one backward call.

The same walk, in PyTorch

import torch

x = torch.tensor([2.0, 3.0], requires_grad=True)
w = torch.tensor([0.5, -1.0], requires_grad=True)

loss = ((w * x).sum() - 1.0) ** 2
loss.backward()

print(loss.item())   # 9.0
print(x.grad)        # tensor([-3.,  6.])
print(w.grad)        # tensor([-12., -18.])

Vanishing and exploding gradients

A deep stack multiplies many local derivatives together. The sigmoid derivative can be as large as 0.25. At that size, ten layers shrink the gradient by a factor of about a million. If each derivative is 2, the same depth multiplies it by a thousand. Depth turns small errors in scale into large ones in either direction.

ReLU keeps the derivative at one on the active path. That property is what made deep networks trainable. A residual connection adds a block’s input to its output, which gives the gradient a path that skips the block entirely. The product then never has to stay near one. Normalisation keeps the activations in a range where the derivatives behave. Gradient clipping scales a gradient down when a single batch produces a spike.

Residual connections are the most important of those fixes. A block computes a correction to its input rather than a replacement. The gradient can then flow through the identity path without passing through every layer. That does not make the gradient correct by itself. It does remove the long product that made deep stacks untrainable.

What gets a gradient

Parameters get gradients because they are leaves with requires_grad set. Inputs can get them too. That is how adversarial examples are built, where an input is changed slightly to fool the model, and how some visualisations work. Activations get gradients during the backward pass and lose them when the graph is freed. Freezing a layer means turning off its gradient, not deleting it.

Gradients accumulate by default. Every backward call adds to .grad until something clears it, which is why the training loop calls zero_grad. That accumulation is also a feature. Summing gradients over several micro-batches is exactly gradient accumulation, and it costs no extra memory for activations.

Detaching is the escape hatch. .detach() returns a tensor that shares the values but not the graph. That is how you stop a gradient from flowing into a target network, a slower copy used as a stable goal. It also works for a replay buffer of past examples, or for a piece of preprocessing. Use it deliberately. A detach in the wrong place produces a parameter that never learns, and no error at all.