Serving diffusion models

The cost is in the loop

Serving a diffusion model is mostly the cost of the sampling loop. One image needs N forward passes through a large network, plus a VAE decode, a safety check and the queue in front of the GPU. A 30-step model at 50 milliseconds a step is 1.5 seconds of compute before the decode. The step count is the first cost knob. It is also the first quality knob, which is why serving and testing cannot be separate teams. The network is the same at every step. Only the noise level changes.

Latency (how long one request takes) and throughput (how many requests finish per second) pull in opposite directions. A batch of eight keeps the GPU busy and lowers the cost per image, but every request in the batch waits for the slowest one. A batch of one is responsive and leaves the hardware idle between requests. The right answer depends on the arrival rate and the service-level objective (the speed the product promises). It does not come from a benchmark run at batch one. Measure both, on the hardware you will actually rent.

Fewer steps, smaller numbers

Distillation trains a student (a smaller model taught to copy the big one) to reach a good sample in four or eight steps instead of thirty or fifty. LCM, SDXL Turbo and the Schnell models are different checkpoints, not a scheduler flag. They were trained for the few-step regime, and using them with a long schedule gives worse results. Solver improvements such as DPM-Solver++ cut the step count without retraining. They cannot beat a model that was distilled for the target.

Precision is the other lever. FP16 halves the weight memory and is the default on modern accelerators. FP8 and INT8 quantisation (storing weights in fewer bits) cut it again at some cost in quality. torch.compile fuses kernels and removes launch overhead, and attention slicing or VAE tiling trade speed for memory. Each change needs a quality check on the same prompt set. A faster pipeline that changes the image is a different product. Quantisation is not free, and the failure usually shows in fine detail rather than in the overall shape.

A turbo pipeline, compiled and offloaded

import torch
from diffusers import AutoPipelineForText2Image

pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/sdxl-turbo",
    torch_dtype=torch.float16,
    variant="fp16",
)
pipe.to("cuda")
pipe.enable_model_cpu_offload()
pipe.unet = torch.compile(pipe.unet, mode="reduce-overhead")

image = pipe(
    prompt="a lighthouse at dusk",
    num_inference_steps=4,
    guidance_scale=0.0,
).images[0]

Memory, offload and tiling

The weights are only part of the footprint. Activations, attention matrices and the VAE decoder’s feature maps all grow with resolution and batch size. A full-attention decode at a large latent size can need more memory than the model itself. Tiling decodes the image in overlapping patches, which caps the peak but repeats the border work. CPU offload moves weights to host memory between steps, which fits the model and adds a transfer to every step. Offload is a fit decision, not a speed decision.

Measure peak memory, not average. A pipeline that fits in 15 GB most of the time will fall over on the one prompt that triggers the largest attention map. Leave headroom for the allocator, the safety model and a second request. If the batch size is chosen from a memory benchmark, run that benchmark at the longest prompt and the highest resolution the product allows.

Serving is an evaluation problem

A production service has an SLO (a target it promises to hit): a p95 latency, a throughput floor and a cost per image. Those numbers belong in the same report as the quality metric, because a cheaper sampler that degrades faces is not a saving. Canary prompts (a small fixed set of test prompts), fixed seeds and pinned model versions make a regression visible before a user sees it. Log the prompt, the seed and the model version together, with consent and retention rules.

Queueing is where latency surprises come from. Utilisation above roughly seventy percent makes waiting time grow faster than the load. A batch that looks efficient at steady state is a latency spike when one request is slow. Autoscale on queue depth rather than CPU. Keep a warm pool to hide the cold start, and cap the queue so overload fails fast instead of timing out. The cheapest image is the one the service can finish. A timeout is a product decision about how long a person will wait.