EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

closed form
A formula that gives a value directly, without stepping through every earlier stage.
denoising
Removing some random corruption from a sample in one step, as the model was trained to do.
diffusion model
A generator trained to turn random noise into data by undoing a chain of small corruptions.
forward process
The fixed chain that adds a little random corruption to a clean sample at each step.
Gaussian
A bell-shaped spread of values around a mean, described by that mean and a width.
guidance
Pushing each sampling step harder toward the text you asked for, trading variety for a closer match.
latent space
A smaller space of numbers where a compressor places images, so the model works on short codes.
noise
Random values added to a sample to corrupt it, drawn from a Gaussian.
noise schedule
The rule that decides how much corruption each step adds, and how fast the real pattern fades.
reverse process
The learned chain that removes corruption step by step, starting from pure randomness.
sampler
The procedure that runs the reverse process to produce a finished sample.
signal-to-noise ratio
How much real structure remains compared with the corruption at a given step.
timestep
One position in the corruption chain, numbered from clean to fully random.
VAE
A model that squeezes an image into a small code and rebuilds it, trained to keep what matters.

Diffusion models from the ground up

Sampling step by step

Ancestral sampling walks the chain with fresh noise at every step; DDIM solves the same model as an ODE. The currency is model calls, not steps.

Part 5 of 10

Advanced17 min readBack to Courses

Ancestral sampling

The simplest sampler walks the chain backwards one step at a time. At each step it predicts the noise, computes the reverse mean, and adds Gaussian noise with the posterior variance. The added noise is not a bug. It is what makes the sample a draw from the distribution rather than a single fixed answer. At the final step the noise is switched off, because there is no next step to correct.

Ancestral sampling needs as many passes as there were training steps, usually a thousand. That is slow, and much of the research on samplers is about doing it in fewer. The first big win was recognising that the reverse process can be written as a differential equation. A differential equation can be solved with a better integrator than Euler with a thousand tiny steps. The gap between a thousand steps and fifty is not a small optimisation. It is the difference between a batch job and an interactive product.

DDIM and the deterministic path

DDIM keeps the same trained model and changes the update. It estimates x_0 at each step. Then it jumps straight to an earlier noise level, using that estimate and the predicted noise. No fresh noise is added. With the random term set to zero, the map from noise to image is deterministic. The same seed gives the same image, and interpolating in the initial noise gives a smooth interpolation in the output.

The deterministic version is also what makes 50-step sampling possible. The update is a coarse integration of an ordinary differential equation. A larger step is therefore a guess rather than a different chain. The error shrinks predictably as steps increase. The same model that was trained with a thousand noise levels can be sampled at fifty. The timesteps you jump between only need to be a subset of the ones it knows. The catch is that the model was trained on every timestep, so choose the subset with the schedule in mind. Do not pick it uniformly out of habit.

Fifty steps with DDIM

import torch
from diffusers import DDIMScheduler, UNet2DModel

unet = UNet2DModel.from_pretrained("google/ddpm-cifar10-32")
scheduler = DDIMScheduler.from_pretrained("google/ddpm-cifar10-32")
scheduler.set_timesteps(50)

sample = torch.randn(4, 3, 32, 32)
for t in scheduler.timesteps:
    noise_pred = unet(sample, t).sample
    sample = scheduler.step(noise_pred, t, sample).prev_sample

print(sample.shape)

Fewer steps, more error

Run a deterministic sampler with 10, 25, 50 and 100 steps on the two-point dataset. Compare how close the samples land to the two modes as the step count rises.

Output
Click "Run" to execute your code

Solvers, order and cost

Every sampler is a trade between the number of model calls and the accuracy of the integration. Euler is first order: error falls in a straight line with step size. Heun and midpoint methods evaluate the model twice per step and are second order, which usually wins when the model is the expensive part. DPM-Solver and its descendants use the semi-linear shape of the probability flow to take larger steps without losing accuracy.

What you pay in is model calls, not steps. A 20-step Heun sampler costs 40 calls, and comparing it with a 40-step Euler sampler is the honest comparison. Papers usually report both. When you ship, measure quality against calls at the step count you can afford. Do not assume that a method tuned for 250 steps still wins at 20.

Compare samplers at equal model calls, not equal steps. A second-order method that uses two calls per step is only a win if it beats the first-order method at the same total budget.

Stochastic or deterministic

The choice changes what the sampler is for. Stochastic ancestral sampling explores the distribution and gives variety from the same start. Deterministic DDIM follows one path and is better for editing, interpolation and inversion. Many production samplers sit between the two with a parameter that controls how much noise is put back, usually called eta. Eta is the dial between them. At zero the update is DDIM, at one it is the ancestral sampler, and in between you get a family that trades variety for stability.

Neither is more correct. The score is the same. Only the step size and the amount of added noise differ. What matters is that you know which one you are running. The same seed and the same prompt will not give the same image across them. Record the sampler, the step count and eta alongside the checkpoint, or a result will not be reproducible.

Follow one sample down

Print the value and the clean estimate at every step of a twenty-step run. The value barely moves for most of the run while the estimate sharpens, then both settle in the last few steps.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What makes ancestral sampling stochastic?