Loss functions

The loss defines the question

Training is optimisation, and optimisation needs a scalar, which is a single number. The loss is that scalar. It compares a prediction with a target and returns a number that is small when the prediction is good. Change the loss and you change what “good” means, even if the architecture and the data stay the same.

Mean squared error averages the squared difference between prediction and target. Squaring punishes large errors more than small ones. That is right when the noise is roughly Gaussian, and wrong when one outlier should not dominate. Mean absolute error averages the absolute difference and is more robust. Its cost is a gradient that is constant in size and undefined at zero.

The gradient of squared error is proportional to the error. A large mistake therefore produces a large correction, and the fit converges quickly when the model is far away. The same property lets an outlier pull the whole fit, because its error dominates the sum. If the outliers are noise, Huber loss switches to an absolute penalty beyond a threshold. That keeps the best of both.

Cross-entropy for classification

A classifier outputs one score per class. Softmax turns those scores into a distribution, and cross-entropy measures the negative log of the probability assigned to the correct class. A confident correct answer costs almost nothing. A confident wrong one costs a lot. That asymmetry is the behaviour a classifier should have.

The softmax and the log combine into a simpler expression. It is the log-sum-exp of the logits, minus the logit of the correct class. Log-sum-exp is a stable way to add exponentials without overflow. That form never divides by a probability and never takes the log of zero. Every framework exposes it as a fused loss that takes logits. The model should therefore not apply softmax itself.

Soft targets are the other reason the log form matters. A soft target is a distribution rather than a single index. Cross-entropy is then minus the sum of target times log probability, and label smoothing fills the target with a small amount for every class. The loss still differentiates cleanly, and the model stops pushing one logit to infinity.

Cross-entropy from logits

import torch
import torch.nn.functional as F

logits = torch.tensor([[2.0, 1.0, -1.0], [0.5, 0.1, 0.4]])
targets = torch.tensor([0, 2])

loss = F.cross_entropy(logits, targets)
print(loss.item())          # mean over the batch

per_example = F.cross_entropy(logits, targets, reduction="none")
print(per_example)          # one value per row

Match the loss to the output

Regression wants squared error or Huber loss. Binary classification wants binary cross-entropy with logits. That is a sigmoid plus a log loss in one stable function. Multiclass wants cross-entropy over the class axis. Multilabel wants one binary loss per label, because the labels are independent and do not sum to one.

The output layer and the loss have to agree. A sigmoid followed by cross-entropy applies the sigmoid twice. A softmax followed by cross-entropy flattens the gradients. Both run, both learn slowly, and neither raises an error. Read the loss documentation once, and note what it expects.

Class weights are the standard fix for imbalance. Multiplying the loss of a rare class by a factor makes a missed rare example more expensive, which moves the boundary. Focal loss goes further. It down-weights examples the model already classifies confidently, so training concentrates on the hard cases. Both change the objective. Both should be evaluated per class rather than by the headline average.

The number is a diagnostic, not a score

A loss value is only comparable across runs with the same reduction, the same preprocessing and the same class balance. A mean loss of 0.3 on one dataset says nothing about a mean loss of 0.3 on another. Report task metrics next to it, such as accuracy, precision, or error in the units of the problem.

Watch the shape of the curve rather than its absolute value. A loss that falls smoothly is healthy. One that spikes may have a learning rate that is too high. One that is flat from the first step usually has a broken gradient or a misaligned label. The first batch is the cheapest place to notice all three.

A NaN is a different signal. It usually comes from a log of zero, a division by a zero variance, or a learning rate that has already diverged. Print the loss and the gradient norm for the first few steps. If the norm grows by an order of magnitude per step, the rate is too high. No amount of loss shaping will fix it.