Segmentation

A mask is a label per pixel

Semantic segmentation assigns a class to every pixel. Its output is not a box but a grid the same height and width as the input. The target is a mask of integer class ids. A mask is a grid of labels, one per pixel. The model produces one score vector per pixel, so a mask with ten classes is a ten-channel image before the argmax.

Instance segmentation goes further and separates two objects of the same class. It needs a mask per object, which is why it is usually built on a detector. Find the objects first, then predict a mask inside each box. Panoptic segmentation combines the two. It labels every pixel and keeps countable things apart.

Shape discipline matters more here than anywhere else in vision. A mask that is one pixel off produces a plausible picture and a wrong score. So does a resize that uses nearest-neighbour for the image and bilinear for the mask. Nearest-neighbour copies the closest pixel, and bilinear blends the four nearest. An argmax over the wrong axis fails the same way. Print the shape of the logits and the mask before you print the metric.

The loss is a classification over pixels

The standard loss is cross-entropy applied at every pixel. The logits are reshaped to (batch × height × width, classes), and the mask to a flat vector of class ids. The loss then averages over the pixels. Nothing about it is special to images. It is the same classifier loss, repeated a few hundred thousand times.

That repetition creates the imbalance problem. A road scene is mostly road and sky, so a model that predicts road everywhere can be 90 percent accurate and useless. The loss is dominated by the common classes. The rare ones, such as a pedestrian or a traffic sign, are the ones that matter.

There are three usual responses. Weight the classes inversely to their frequency, sample patches that contain rare classes, or use a loss that measures overlap rather than per-pixel accuracy. Dice loss measures overlap between the predicted mask and the target. It is common in medical imaging, where the foreground can be a few hundred pixels in a large scan.

Cross-entropy on a segmentation output

import torch
from torch import nn
from torchvision.models.segmentation import deeplabv3_resnet50

model = deeplabv3_resnet50(weights=None, num_classes=21)
images = torch.randn(2, 3, 256, 256)
masks = torch.randint(0, 21, (2, 256, 256))

logits = model(images)["out"]                  # (2, 21, 256, 256)
loss = nn.functional.cross_entropy(logits, masks)
print(tuple(logits.shape), round(loss.item(), 3))

Encoders, decoders and skips

An encoder-decoder network first reduces the image to a coarse feature map. It then expands the map back to full resolution. The encoder is the same stack of convolutions and pooling used for classification. The decoder replaces pooling with upsampling, either by interpolation or by a transposed convolution. A transposed convolution is a learned upsampling step.

The problem with a plain decoder is that detail is lost on the way down. A skip connection copies the high-resolution features from an encoder stage to the matching decoder stage. The decoder can then combine coarse semantics with fine edges. U-Net is the canonical shape: a contracting path, an expanding path, and a concatenation across every level.

Stride is the other knob. Predicting at half resolution and upsampling the logits is cheaper and often accurate enough. Predicting at full resolution costs memory but keeps thin structures. The output stride belongs in the model configuration. It decides both the cost and the smallest object the model can resolve.

Reading a segmentation report

Report intersection over union per class, then average it. Pixel accuracy is the number that looks best and says least, because the background class usually owns most of the image. Mean IoU treats every class equally. That is what you want when the rare class is the point.

A confusion matrix over pixels is the fastest way to see what is happening. The diagonal is correct, and each off-diagonal cell is one class being mistaken for another. A model that confuses sidewalk with road is making a different mistake from one that confuses pedestrian with sign. The matrix tells them apart before you open an image.

Boundaries deserve their own number. A mask can have a high IoU and a ragged edge. The edge is what a user sees when the mask is composited over a video. Trimap evaluation, boundary F1 and a simple band around the ground-truth contour all measure the part that the aggregate hides. Report one of them alongside the mean.