EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

attention
A weighted blend of value rows, with weights decided by how well each query matches each key.
attention map
The table of scores showing how much each position attended to every other position.
causal mask
A rule that blocks each position from seeing later positions, so predictions can only use the past.
dimension
One axis of a list or grid of numbers; its length is how many values fit along that axis.
distribution
The pattern of how often each possible value appears, or how likely each one is.
dot product
Multiply matching numbers in two lists and add the results; a large total means the lists point the same way.
embedding
A learned list of numbers that stands for a token, with similar tokens placed near each other.
GPU
A processor built for many small calculations at once, which is why matrix work runs well on it.
gradient
A list of slopes, one per number, saying how much each nudge changes the result.
hard pointer
A choice that sends all the weight to one position, instead of spreading it across several.
head
One parallel slice of attention, with its own projections; a layer runs many so it can track different relations at once.
key
What a position advertises it can offer, compared against every query to decide who receives attention.
key axis
The direction in the key vectors whose width decides how far the scores are divided down.
layer
One stage of a model that reads the running vector, transforms it and passes it on.
layer norm
A per-position rescale that keeps the residual stream at a steady size before the next block.
logits
The raw scores a model produces before softmax turns them into probabilities.
matrix
A grid of numbers arranged in rows and columns, used to transform a list into another list.
negative infinity
A value smaller than every real number, used to mark a score that must never be chosen.
one-hot
A vector with a single one and zeros elsewhere, meaning all the weight sits on one choice.
output projection
The learned matrix applied after attention, which turns the mixed values back into the model's width.
parameter
A number inside a model that training adjusts, rather than an input it reads.
positional encoding
Extra numbers added to token embeddings so the model can tell which position each one came from.
projection
A learned matrix that turns a list of numbers into a different list, often shorter.
query
What a position is looking for, compared against every key to decide where to take information from.
residual stream
The running vector each block reads from and adds its output back into.
retrieval layer
A layer whose job is to pull specific stored information into the current position.
saturating
A score so large that the softmax gives almost all the weight to one position.
scale
The step that divides the scores by the square root of the key width, so they stay in a workable range.
score matrix
The grid holding every query's score against every key, one row per position.
soft alignment
Matching by degree rather than by a single choice, so several positions can contribute a little.
softmax
A rule that turns a list of scores into positive weights that sum to one.
token
A chunk of text — a word, part of a word, or a character — that the model treats as one unit.
value
The content a position passes on when attention selects it.
variance
The average squared distance of values from their average, so a spread-out set scores higher.
vector
An ordered list of numbers, where each position is one measured property.
weighted sum
Adding values together with different shares, where the shares come from the attention scores.
weights
The numbers a model multiplies inputs by before adding the results together.

Transformers from the ground up

Positional information

Attention is permutation-invariant, so position has to be injected. Sinusoidal encodings, learned tables and rotary embeddings are three answers to the same problem.

Part 6 of 10

Advanced16 min readBack to Courses

Attention has no order

Shuffle the tokens of a sequence and shuffle the attention rows and columns to match. The output is the same, up to that shuffle. Attention compares content with content. Nothing in the dot product knows whether a token came first or fifth. That is useful for sets. It is a bug for language, where “dog bites man” and “man bites dog” contain the same tokens.

The fix is to add position information to the token vectors before the first block. The model then sees a vector that says both what the token is and where it sits. Every later layer inherits that signal through the residual stream. The code therefore has to be applied only once, at the input.

Absolute encodings

The original transformer used a fixed sinusoidal table. Sine and cosine waves fill the vector. Each dimension oscillates at a different rate: fast for the early pairs, slow for the later ones. The pattern is fixed in advance and needs no weights. It gives every position a unique fingerprint that the model can learn to read.

A learned table is the simpler alternative. It is an embedding matrix with one row per position, trained like the token embeddings. It fits the training data well. It cannot extrapolate beyond the longest sequence it saw, because there is no row for position 4,097. A sinusoidal table can in principle reach further. A model trained on short sequences still rarely generalises to long ones without other changes.

The sinusoidal table has a useful property. A shift in position is a linear change to the code. That does not make the model relative on its own. It does give the weights a consistent pattern to learn. The rates are spaced geometrically. The fastest pair completes a cycle every few positions. The slowest pair changes very little across a whole sequence.

A learned position table

import torch
from torch import nn

positions = torch.arange(seq_len).unsqueeze(1)
table = nn.Embedding(max_positions, d_model)

x = token_embeddings + table(positions)
print(x.shape)  # (batch, tokens, d_model)

A fixed position fingerprint

Print the encodings for four positions, then compute the dot product between position zero and each of the others and describe how the similarity decays.

Output
Click "Run" to execute your code

Relative and rotary

Absolute codes tell a token where it is. Relative methods tell it how far away another token is. That is often the more useful fact. Rotary position embedding, or RoPE, takes the query and key vectors. It rotates each pair of dimensions by an angle that depends on the position. The dot product of a rotated query and key then depends on their offset. It does not depend on their absolute positions.

The rotation is applied inside attention, not added to the input. The residual stream therefore stays free of position vectors. RoPE is the default in many current language models. It is often combined with a scaling rule that stretches the rates for longer contexts. ALiBi takes a different route. It subtracts a distance penalty from the attention scores, with no position vectors at all.

RoPE is usually applied per head. It comes after the query and key projections and before the dot product. That placement matters. The values are not rotated, so the content that gets mixed is unchanged. The residual stream never sees a position vector. The rotation is a property of the comparison, not of the vector itself.

When you pack several documents into one sequence, reset or offset the position ids at each boundary. A model that sees document two starting at position 900 has been told something false about where its first token sits.

What position does not fix

A position code gives the model a coordinate. It does not guarantee that the model will use distance correctly. Attention can still spread across a long context. A model trained on short sequences may ignore positions far outside its experience. Extending the context window usually needs more than a bigger table. It needs training on longer inputs, or a deliberate stretch of the rates.

Position also depends on the task. Copying a token from earlier is a relative operation. Answering a question about the third paragraph is closer to an absolute one. Models learn both. A change in context length often shows up here first, as a quiet drop in quality rather than an error.

Long-context testing is its own discipline. A model can pass a needle-in-a-haystack test and still fail to use the middle of a long document. Finding a fact and reasoning with it are different demands. Position codes set the stage. The test has to check whether the model reads the whole stage.

Rotate instead of add

Run the relative scores, then confirm that shifting both positions by the same amount leaves the dot product unchanged.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

Why does a transformer need positional information?