Guidance and conditioning

Conditioning is an extra input

A text-to-image model is a diffusion model whose denoiser also sees a description. The description is encoded once into a sequence of vectors. Every denoising step then attends to them through cross-attention (a way for one part of the network to look at another input). Nothing about the diffusion maths changes. The model still predicts noise, and the sampler still walks the chain. The condition only shifts which noise the model predicts at each step. The encoder is usually a pretrained text model, frozen or lightly tuned. Its output is the only thing the denoiser knows about the prompt.

Training is the same objective with pairs of image and caption. The caption is dropped at random for a fraction of examples, usually ten percent, and replaced with an empty one. That single trick is what makes classifier-free guidance possible later. The same weights learn both the captioned and the empty prediction. Without it you would need a separate classifier to guide with.

Classifier-free guidance

At sampling time you run the model twice: once with the caption and once with the empty one. Then combine the two predictions. Take the empty-prompt noise and add a scale times the gap between the captioned and empty predictions. A scale of one gives the ordinary captioned model. Above one, the gap is amplified, and the sample moves further in the direction the caption describes.

The scale is not a probability and not a temperature. It multiplies a gap in noise predictions, and the result is a sharper, less varied distribution. Small values add detail. Large values produce saturated colours, clipped highlights and repeated shapes. The useful range depends on the model, usually somewhere between five and twelve. Tune it with the sampler and step count you will ship. The scale is applied to the gap, so it has no effect when the two predictions agree. Guidance does little at low noise for that reason. There the sample already meets the condition.

Guidance in a pipeline

import torch
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16,
).to("cuda")

image = pipe(
    prompt="a glass bottle on a windowsill, morning light",
    negative_prompt="blurry, low contrast",
    num_inference_steps=30,
    guidance_scale=7.5,
).images[0]
image.save("bottle.png")

What guidance actually does

Guidance sharpens the distribution that the caption describes. In the language of scores, it adds a multiple of the captioned score minus the empty one. That gap is the score of the ratio of the two densities. The ratio is large where the caption is more likely, so the sampler is pushed toward those regions. The price is that the sampler no longer follows the true score of any distribution. The further you push, the more the sample leaves the data manifold (the region where real data lives).

The same mechanism covers negative prompts: guide away from a second condition instead of toward the empty one. It also covers classifier guidance, the older method that used a separate classifier’s gradient. Classifier-free guidance won because it needs no extra model. It also won because the empty branch is trained on the same data, so the gap is well behaved. A checkpoint trained without caption dropping has no empty branch, so it cannot be guided this way at all.

Other kinds of conditioning

Text is only one channel. ControlNet adds a parallel network that takes an edge map, a depth map or a pose, and injects its features into the denoiser. The base model stays frozen. Inpainting masks the known region and keeps it clean at every step. Image-to-image starts the chain partway, at a timestep that matches the requested strength, so the output keeps the layout of the input. Each method has its own training data and its own failure modes, but they share the same interface to the sampler.

All of these are the same idea with different plumbing: give the denoiser more information and it predicts a different noise. The sampler does not need to know which kind of condition it is. A new conditioning method can therefore ship as a new network or a new input channel without touching the sampling loop. That is also what makes the loop the part worth understanding well.