The idea of adding noise

Destruction is easy

A generative model has to produce samples from a distribution it has only seen examples of. A distribution is the set of all possible outputs, with a weight for how likely each one is. Diffusion takes the hardest part of that job, which is jumping from nothing to a structured sample. It replaces that jump with many small problems that are almost trivial. Take a data point, add a little Gaussian noise (noise drawn from a bell curve), and repeat. After enough steps the structure is gone, and what remains is a standard normal sample (pure noise with mean zero and variance one). The model learns to undo those steps one at a time. Sampling is that path walked backwards.

Each step is small enough that a network only has to estimate a local correction, not invent an image. A thousand of those corrections add up to a sample. For that reason, the literature calls the method learning to denoise rather than learning to generate. Generation is denoising that starts at the maximum noise level. The same machinery also gives you a likelihood, a latent space (a smaller representation the model works in) and a way to condition on text. A likelihood says how probable the data is under the model. That range is why one idea became the basis of most image models.

Noise as a coordinate

Any image can be placed on a scale that runs from clean to pure noise. The forward process assigns every data point a family of noisy versions, indexed by a timestep t. The model’s job is to learn, at every point on that scale, which direction leads back toward data. That direction is the score: the gradient (the slope) of the log density with respect to the input. If you know the score at every noise level, you can walk from noise to data without ever knowing the density itself.

The score is not something we can compute for a photograph, because we do not know the distribution. One identity saves us. For Gaussian noise, the score of the noisy distribution is the conditional expectation of the noise that was added, with a sign and a scale. Predicting the noise and predicting the score are the same task. Every training objective in this course is a variation on that one fact.

Add noise with a scheduler

import torch
from diffusers import DDPMScheduler

scheduler = DDPMScheduler(num_train_timesteps=1000, beta_schedule="linear")
image = torch.randn(4, 3, 64, 64)
noise = torch.randn_like(image)
timesteps = torch.tensor([0, 250, 500, 750])

noisy = scheduler.add_noise(image, noise, timesteps)
print(noisy.shape)
print(scheduler.alphas_cumprod[timesteps])

Why not generate in one shot

A GAN asks a network to map noise to data in a single jump. A GAN is a generator trained against a critic that tries to spot fakes. That can work well, but the loss is a moving target. Training is a balance between two networks, and it can fail in confusing ways. Diffusion asks for a much weaker thing at every step: given a slightly noisy sample, estimate the noise that was added. The target is fixed, the loss is mean squared error, and the training curve is boring in the best sense.

The price is speed. A single sample needs one forward pass per step, and a thousand steps is a thousand passes. Much of the engineering in later chapters is about buying those passes back. Better schedules help. So do deterministic samplers, and so does running diffusion in a compressed latent space rather than in pixels.

What the network predicts

Three parameterizations show up everywhere. The model can predict the noise eps that was added, the clean sample x_0, or the velocity v. Velocity is a fixed mix of the other two. They carry the same information, so a model trained on one can be converted to another. What changes is how the loss weights the timesteps, and how well the numbers behave at the extremes.

Predicting eps is the original choice and still the default. At t near zero the input is almost clean, so eps is almost pure noise and hard to estimate. At t near the end the input is almost noise, so x_0 is the hard one. Velocity splits the difference, and it is popular with cosine schedules (schedules that keep more signal for longer), where both ends matter. Pick one, write it down, and convert on purpose rather than mixing conventions between training and sampling.