Vision transformers

An image as a sequence

A transformer expects a sequence of vectors. A vision transformer, or ViT, makes one by cutting the image into fixed-size patches. It flattens each patch and projects it with a learned linear layer. A token is one vector in that sequence. A 224 by 224 image with 16 by 16 patches becomes 196 tokens, plus one special token used for the classification output.

Nothing in that operation knows about adjacency. The patch order is arbitrary until a positional embedding is added. A positional embedding is a learned vector that marks where each patch came from. The model has no convolution’s built-in assumption that nearby pixels belong together. The positional embedding restores a sense of place, and it is learned rather than fixed in the original ViT.

The patch size is the main design choice. Smaller patches give more tokens and finer detail at quadratic cost. Larger patches are cheaper and coarser. A 32 by 32 patch turns the image into 49 tokens, which is fast and loses small objects. Most production models sit at 14 or 16.

Attention in one line

Self-attention turns each token into a query, a key and a value. A query asks what to look for, a key says what a token offers, and a value is the content that gets mixed. It scores every query against every key, normalises the scores with a softmax, and mixes the values. The scaling by the square root of the head dimension keeps the logits from saturating as the dimension grows.

Multi-head attention runs several of these in parallel with smaller projections, then concatenates the results. Each head can specialise. One tracks colour, another tracks a repeated texture, and another relates distant parts of an object. The concatenation is projected back to the model width. The head dimension is usually the width divided by the number of heads, so the total cost stays close to one full-width projection.

The cost is the problem. Every token attends to every other, so the attention matrix is quadratic in the number of tokens. Doubling the resolution quadruples the tokens and multiplies attention cost by sixteen. High-resolution vision transformers therefore use windowed attention, sparse attention, or a convolutional stem.

A ViT in timm

import timm
import torch

model = timm.create_model("vit_base_patch16_224", pretrained=False, num_classes=10)
images = torch.randn(4, 3, 224, 224)

print(model(images).shape)              # (4, 10)
print(model.patch_embed.num_patches)    # 196

Why data and regularisation matter

A convolutional network is born with the right priors: locality, translation equivariance, and a hierarchy that grows with depth. A transformer has none of them, so it has to learn them from data. Trained on a small dataset, a ViT underperforms a ResNet of similar size. ResNet is a convolutional network. Trained on hundreds of millions of images, the ViT overtakes it.

DeiT showed that distillation from a convolutional teacher closes much of the gap on ImageNet alone. Distillation trains one model to copy another. It also showed that strong augmentation is not optional. Mixup, cutmix, random erasing and RandAugment are part of the recipe, not decoration. Dropout and stochastic depth keep a deep transformer from memorising. Stochastic depth randomly skips whole blocks during training.

Fine-tuning follows the same logic. It means starting from pretrained weights and training further on your own data. A pretrained ViT transfers well with a small learning rate on the head and a smaller one on the backbone. The backbone is the main feature extractor, before the classifier. Training from scratch on a few thousand images is usually a mistake. The pretraining dataset and the resolution it used belong in the model card, because both affect what the model can see.

Hybrids and windows

Pure attention over a full image is expensive, so most modern vision transformers restrict it. Swin computes attention inside shifted windows and merges tokens between stages. A window is a local group of tokens. That restores a hierarchy and linear cost in the number of pixels. Convolutional stems do the same job by starting with a few downsampling layers.

A hybrid is often the pragmatic choice: a convolutional backbone for local features and a transformer for global context. The convolutional part handles edges and textures cheaply. Attention handles the relationships that a small kernel cannot reach. The boundary between them is only a tensor, and either side can be replaced.

The evaluation does not change. Accuracy, per-class recall and robustness to resolution and corruption still decide whether the model is useful. A transformer that wins on clean ImageNet and loses on a blurred photo has not solved the problem the product has. Test the corruptions the deployment will actually see.