Detection and boxes

Boxes and coordinates

A box is four numbers. The two conventions are corner form, (x1, y1, x2, y2), and centre form, (cx, cy, width, height). Corner form is convenient for clipping and intersection. Centre form is convenient for offsets and anchors. Most bugs in a detector come from mixing the two. Most of those bugs produce a box that is plausible enough to pass a visual check.

The origin is the top left, so y increases downward. A box that extends past the image edge has to be clipped before it is compared with a ground-truth box. Ground truth means the hand-labelled answer. Without clipping, the area is wrong. Normalised coordinates divide by width and height. They let a model share a head across resolutions, at the cost of a conversion at both ends.

Clipping is not optional. A predicted box can have negative coordinates or extend past the image. Its area is then wrong for IoU and for the loss. Clamp the corners to the image bounds before comparing. Keep the unclipped version only if the loss needs the gradient through the edge.

Intersection over union

IoU, short for intersection over union, is the area of the intersection divided by the area of the union. Two identical boxes score one, two disjoint boxes score zero, and a box half the size of another and completely inside it scores 0.5. It is scale invariant, which is why one threshold works for a small object and a large one.

A prediction is usually counted as correct when its IoU with a ground-truth box is at least 0.5. Stricter thresholds are used to reward localisation. The intersection is computed on the overlapping rectangle. Take the larger of the two left edges and the smaller of the two right edges, and do the same for the vertical axis. If either side is negative, there is no overlap.

IoU has a blind spot. Two boxes that do not overlap have an IoU of zero, regardless of how far apart they are. The gradient is flat, and the model gets no signal to move them together. GIoU adds a penalty for the gap, DIoU adds a centre-distance term, and CIoU adds aspect ratio. They are refinements of the same idea.

IoU and NMS in torchvision

import torch
from torchvision.ops import box_iou, nms

boxes = torch.tensor([[0, 0, 10, 10], [5, 5, 15, 15]], dtype=torch.float)
scores = torch.tensor([0.9, 0.8])

print(box_iou(boxes, boxes))
print(nms(boxes, scores, iou_threshold=0.5))   # tensor([0])

Anchors, strides and heads

A one-stage detector divides the image into a grid and predicts a set of boxes at each cell. The boxes are relative to anchors. An anchor is a reference box of a fixed size and shape. The head, the small network that produces predictions, outputs an objectness score, a class distribution and four offsets that refine the anchor. Objectness is the score that says a box contains an object. A two-stage detector first proposes regions and then classifies and refines them. It is slower and usually more accurate on small objects.

Modern detectors are often anchor-free. They predict a centre and a size directly, or a distance to each side of the box. The vocabulary changes; the output is still a set of boxes with scores. What matters for the rest of the pipeline is that the boxes are in the same coordinate system as the ground truth. Duplicates also have to be removed.

Training a detector is an assignment problem before it is a regression problem. Regression means predicting a number rather than a class. Each ground-truth box must be matched to one or more anchors or predictions, and the rest are negatives. The matching rule decides what the model is rewarded for. A sloppy rule produces a detector that localises well and classifies poorly, or the reverse.

Measuring a detector

Precision is the fraction of returned boxes that are correct. Recall is the fraction of real objects that were found. Raising the confidence threshold trades recall for precision. Average precision summarises that trade for one class by integrating the precision-recall curve. Mean average precision averages it over classes.

COCO reports mAP at several IoU thresholds and separately for small, medium and large objects. A single number hides the failure that matters most. A detector can be excellent on large objects and blind to small ones. Look at the per-size numbers and at a few images before trusting the summary.

Average precision is computed per class by sweeping the confidence threshold. At each step it measures precision and recall, then integrates the curve. The details differ between VOC and COCO, so a number is only comparable within one convention. State the dataset, the IoU threshold and the convention whenever you report mAP.