Evaluating generated images

A sample is not a score

A diffusion model can make one great image and still be a bad model. Testing it asks two different questions. Does the distribution of samples match the data? Does each sample match its prompt? A distribution is the set of outputs a model can make, with a weight for how often each appears. Metrics on the whole set answer the first question. Per-image metrics and human ratings answer the second. A demo answers neither, because it shows the best sample the author could find. A benchmark is a claim about a whole set, and one sample is not evidence about a set.

The two questions can disagree. A model that makes the same perfect face for every prompt has excellent fidelity (how real each image looks) and no variety. A model that makes a different blurry shape for every prompt has variety and no fidelity. A single number cannot separate those failures. A credible report gives at least one distribution metric, one alignment metric and a human check. A report that shows only one of the three has chosen which failure to hide. Report the prompts, the seeds and the number of samples, or the number cannot be checked.

FID and what it assumes

FID embeds real and generated images with an Inception network. It fits a multivariate Gaussian (a bell curve in many dimensions) to each set of features, and measures the Fréchet distance between them. Lower is better. The metric rewards features that are close in mean and covariance, which captures both quality and variety in one number. It is cheap enough to run often and sharp enough to catch a broken sampler. The feature extractor is part of the metric, not a neutral observer.

It also assumes a lot. The Gaussian is a rough fit. The Inception features were trained to sort ImageNet photos, not your domain. The estimate depends on the sample size. You cannot compare FID across extractors, image sizes, preprocessing or sample counts. KID removes some of the sample-size bias by using a kernel two-sample test, and precision and recall split the question in two. Report the extractor and the number of images with the score.

FID with torchmetrics

import torch
from torchmetrics.image.fid import FrechetInceptionDistance

fid = FrechetInceptionDistance(feature=2048, normalize=True)
fid.update(real_images, real=True)
fid.update(fake_images, real=False)

score = fid.compute()
print(float(score))

Precision, recall and two ways to fail

Precision asks whether a generated sample looks like it belongs to the real data. Recall asks whether the set of generated samples covers the real data. A mode-dropping model that makes only the easiest mode has high precision and low recall. A mode is one cluster of the data. A model that spreads noise over the whole space has high recall and low precision. FID mixes both into one distance, which is handy and hard to debug. Neither number is good or bad on its own. The pair describes a shape.

Per-image alignment is a separate axis. CLIPScore compares an image vector with a text vector. It tracks human judgements of prompt adherence, but it is a proxy trained on captions, and text in the image can fool it. Use it with a human check, and always test on prompts the model has not seen. A prompt set that leaks into training is a benchmark that measures memorisation.

Human evaluation and memorisation

Human review is still the reference. A pairwise preference study is more reliable than an absolute score, because people are better at choosing than at rating. Fix the prompts, the seeds and the display order. Collect enough comparisons for a confidence interval, and report how far the raters agree. A panel of ten colleagues is a demo, not a study. Agreement below chance is a sign the question or the interface is broken.

Memorisation is the failure that a distribution metric cannot see. A model that copies a training image exactly has perfect fidelity and cannot generalise. Check nearest neighbours in feature space. Look for near-duplicates, and plant canary images in the training set to see if they come back. Leakage works the other way too: a benchmark whose images appear in training measures recall, not generation. Keep the test set out of the training pipeline and say so. A memorised sample is a privacy incident as well as a test failure.