Diffusion models from the ground up
Add noise, learn to undo it, and sample
- Advanced
- 10 chapters
- 164 min read
How a diffusion model actually works: the forward chain and its closed form, the reverse Gaussian and the noise-prediction loss, schedules and signal-to-noise ratios, ancestral and deterministic samplers, classifier-free guidance, latent diffusion, and a capstone one-dimensional sampler built from a table and checked against the exact answer.
What You'll Learn
- Explain why gradual corruption is easier to learn than one-shot generation
- Derive the closed form for a noisy sample at any timestep
- Write the reverse update and test it with the true noise
- Read a schedule by its signal-to-noise ratio and half-signal point
- Compare samplers by model calls and step count
- Apply classifier-free guidance and explain its failure modes
- Explain what a latent autoencoder changes and what it costs
- Build a one-dimensional diffusion sampler and validate it against the exact denoiser
Prerequisites
- Comfortable with Python functions, lists and loops
- Comfortable with the idea of a probability distribution and a mean
- No GPU, deep learning framework or API key is needed
Vocabulary
The terms the chapters assume, in plain words.
- closed form
- A formula that gives a value directly, without stepping through every earlier stage.
- denoising
- Removing some random corruption from a sample in one step, as the model was trained to do.
- diffusion model
- A generator trained to turn random noise into data by undoing a chain of small corruptions.
- forward process
- The fixed chain that adds a little random corruption to a clean sample at each step.
- Gaussian
- A bell-shaped spread of values around a mean, described by that mean and a width.
- guidance
- Pushing each sampling step harder toward the text you asked for, trading variety for a closer match.
- latent space
- A smaller space of numbers where a compressor places images, so the model works on short codes.
- noise
- Random values added to a sample to corrupt it, drawn from a Gaussian.
- noise schedule
- The rule that decides how much corruption each step adds, and how fast the real pattern fades.
- reverse process
- The learned chain that removes corruption step by step, starting from pure randomness.
- sampler
- The procedure that runs the reverse process to produce a finished sample.
- signal-to-noise ratio
- How much real structure remains compared with the corruption at a given step.
- timestep
- One position in the corruption chain, numbered from clean to fully random.
- VAE
- A model that squeezes an image into a small code and rebuilds it, trained to keep what matters.
Course Content
- 01The idea of adding noiseNot started
Instead of learning to generate in one jump, a diffusion model learns to undo a gradual corruption. Each step is a small denoising problem, and sampling is the path walked backwards.
- 02The forward processNot started
The forward chain adds a little noise at every step, and Gaussian noise composes. That gives a closed form for any timestep, which is what makes training cheap.
- 03The reverse processNot started
The reverse step is a Gaussian whose mean depends on the predicted noise. Training is a mean squared error on that prediction, and one step of algebra recovers the clean sample.
- 04Noise schedulesNot started
The betas decide where the model spends its capacity. A good schedule spreads the work evenly across signal-to-noise ratios instead of destroying the signal early.
- 05Sampling step by stepNot started
Ancestral sampling walks the chain with fresh noise at every step; DDIM solves the same model as an ODE. The currency is model calls, not steps.
- 06Guidance and conditioningNot started
A conditional model predicts noise given a description. Classifier-free guidance amplifies the difference between the conditional and unconditional predictions, trading diversity for adherence.
- 07Latent diffusionNot started
Run diffusion inside a compressed autoencoder latent instead of pixels. The maths is unchanged; the tensors are smaller and the expensive attention becomes affordable.
- 08Evaluating generated imagesNot started
A good sample is not a good model. FID, precision and recall describe a distribution; CLIPScore and people describe a single image. A credible report needs both, with the prompts and the sample count attached.
- 09Serving diffusion modelsNot started
An image is a loop of network calls, a decode and a queue. Serving it well means choosing steps, precision, batch size and memory together, against a latency target and a cost per image.
- 10Capstone: a one-dimensional diffusion samplerNot started
Fit a denoiser on two points, then run the reverse chain on it. The whole method fits in one file, and every number can be checked against the exact answer.