Deep learning from the ground up
Networks built from numbers you can check
- Advanced
- 10 chapters
- 160 min read
What a neural network actually computes: nested lists with a shape, a forward pass that bends, a loss that turns a prediction into one number, and a backward pass that is the chain rule in reverse. Then the parts that make it train — optimisers, regularisation, batching, recurrent layers and the gradients that travel through time, mixed precision and the memory budget of a GPU step — and a capstone autograd engine that fits XOR.
What You'll Learn
- Read a tensor’s shape and keep the axes straight
- Build a dense layer and explain why the nonlinearity matters
- Choose a loss that matches the output and compute it stably
- Derive and check gradients by hand
- Compare SGD, momentum and Adam on the same problem
- Diagnose overfitting and apply the right regulariser
- Write a training loop with batches, evaluation and accumulation
- Carry a hidden state through a sequence and clip its gradients
- Explain fp16, bf16 and the memory cost of a training step
- Build a scalar autograd engine and train a small network
Prerequisites
- Comfortable with Python lists, functions and classes
- Basic linear algebra: vectors, matrices and dot products
- No ML framework is needed — every example is plain Python
Vocabulary
The terms the chapters assume, in plain words.
- activation
- A rule applied to each number in a layer that lets the network bend, rather than only scale and add.
- autograd
- Software that records operations during a forward pass and later computes their gradients automatically.
- backpropagation
- Working backwards through a network to find how each weight affects the loss, by multiplying local slopes.
- batch
- A group of examples processed together in one step, averaged into a single update.
- epoch
- One full pass over the training examples.
- forward pass
- Running inputs through a network to produce outputs, before any error is measured.
- gradient
- The direction and steepness in which a value rises fastest; training moves weights the opposite way.
- learning rate
- How far each update moves the weights; too large overshoots, too small crawls.
- loss
- One number saying how wrong the current predictions are, which training tries to reduce.
- optimiser
- The procedure that uses gradients to decide how each weight changes.
- overfitting
- When a network memorises accidental details of its training examples and does worse on new ones.
- regularisation
- Any penalty or limit that discourages a network from chasing noise in its training examples. The aim is better performance on new data.
- tensor
- A nested list of numbers with a shape that says what each axis means.
Course Content
- 01Tensors as nested listsNot started
A tensor is a nested list plus a shape. Every layer is a shape transformation, and most deep learning bugs are shape bugs wearing a numeric costume.
- 02The forward passNot started
A layer is a matrix multiply, a bias and a nonlinearity. Stack them and the nonlinearity is the only reason the stack is more than one big matrix.
- 03Loss functionsNot started
A loss turns a prediction into one number the optimiser can lower. Mean squared error and cross-entropy are the two you will use, and both have a careful form.
- 04Backpropagation by handNot started
Backpropagation is the chain rule applied in reverse order, with each node passing its local gradient back to the nodes that fed it. Nothing more.
- 05OptimizersNot started
Gradient descent says which way is downhill. The optimiser decides how far to step, and that decision is most of what separates a run that converges from one that does not.
- 06Overfitting and regularisationNot started
A model that memorises its training set has learned the noise. Regularisation is any constraint that makes the easy answer harder, and validation is how you know it worked.
- 07Batching and the training loopNot started
An epoch is a pass over the data, a step is one update. The loop that connects them is short, and every line in it is a place a run can go quietly wrong.
- 08Sequence modelsNot started
A recurrence applies one rule at every step and carries a state between them. That sharing is what handles variable length, and it is also why gradients have to travel so far.
- 09GPUs and mixed precisionNot started
A GPU is a throughput machine, and a float is a budget of range and resolution. Training in mixed precision is the art of spending that budget where it does not change the answer.
- 10Capstone: a tiny autogradNot started
A scalar autograd engine is a hundred lines: a value, its gradient, the operation that made it, and a reverse walk over the graph. Training a network then needs nothing else.