EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

attention
A weighted blend of value rows, with weights decided by how well each query matches each key.
attention map
The table of scores showing how much each position attended to every other position.
causal mask
A rule that blocks each position from seeing later positions, so predictions can only use the past.
dimension
One axis of a list or grid of numbers; its length is how many values fit along that axis.
distribution
The pattern of how often each possible value appears, or how likely each one is.
dot product
Multiply matching numbers in two lists and add the results; a large total means the lists point the same way.
embedding
A learned list of numbers that stands for a token, with similar tokens placed near each other.
GPU
A processor built for many small calculations at once, which is why matrix work runs well on it.
gradient
A list of slopes, one per number, saying how much each nudge changes the result.
hard pointer
A choice that sends all the weight to one position, instead of spreading it across several.
head
One parallel slice of attention, with its own projections; a layer runs many so it can track different relations at once.
key
What a position advertises it can offer, compared against every query to decide who receives attention.
key axis
The direction in the key vectors whose width decides how far the scores are divided down.
layer
One stage of a model that reads the running vector, transforms it and passes it on.
layer norm
A per-position rescale that keeps the residual stream at a steady size before the next block.
logits
The raw scores a model produces before softmax turns them into probabilities.
matrix
A grid of numbers arranged in rows and columns, used to transform a list into another list.
negative infinity
A value smaller than every real number, used to mark a score that must never be chosen.
one-hot
A vector with a single one and zeros elsewhere, meaning all the weight sits on one choice.
output projection
The learned matrix applied after attention, which turns the mixed values back into the model's width.
parameter
A number inside a model that training adjusts, rather than an input it reads.
positional encoding
Extra numbers added to token embeddings so the model can tell which position each one came from.
projection
A learned matrix that turns a list of numbers into a different list, often shorter.
query
What a position is looking for, compared against every key to decide where to take information from.
residual stream
The running vector each block reads from and adds its output back into.
retrieval layer
A layer whose job is to pull specific stored information into the current position.
saturating
A score so large that the softmax gives almost all the weight to one position.
scale
The step that divides the scores by the square root of the key width, so they stay in a workable range.
score matrix
The grid holding every query's score against every key, one row per position.
soft alignment
Matching by degree rather than by a single choice, so several positions can contribute a little.
softmax
A rule that turns a list of scores into positive weights that sum to one.
token
A chunk of text — a word, part of a word, or a character — that the model treats as one unit.
value
The content a position passes on when attention selects it.
variance
The average squared distance of values from their average, so a spread-out set scores higher.
vector
An ordered list of numbers, where each position is one measured property.
weighted sum
Adding values together with different shares, where the shares come from the attention scores.
weights
The numbers a model multiplies inputs by before adding the results together.

Transformers from the ground up

Attention by hand

Attention is three projections and a weighted average. Every token asks a question, every token advertises what it knows, and a softmax decides who answers.

Part 2 of 10

Advanced17 min readBack to Courses

Three vectors per token

Each token’s vector is projected three ways. The query is what this position is looking for. The key is what this position can offer. The value is what gets passed on if the offer is accepted. The projections are learned matrices. The same input can therefore ask different questions in different layers.

A sequence of n tokens gives three matrices of n rows. Attention compares every query with every key. That is where the n × n score matrix comes from. Nothing in the operation is sequential. All scores can be computed at once. So the layer runs well on a GPU.

The projections are the only learned parameters in this calculation. The rest is arithmetic: a dot product, a division by the square root of the key width, a softmax and a weighted sum. So an attention block is easy to implement and hard to interpret. The complexity lives in the learned matrices, not in the dozen lines that apply them. A correct implementation tells you nothing about what the weights mean.

Scores, scale, softmax

A score is a dot product between one query and one key. A dot product multiplies matching entries and adds the results. Dot products grow with the dimension of the vectors. A large score makes the softmax nearly one-hot, meaning one weight takes almost all the mass. The other positions then get almost no gradient. Dividing by the square root of the key dimension keeps the variance roughly constant as the width grows.

The scaled scores go through a softmax along the key axis. Each row becomes a distribution over the tokens it is allowed to see. The output for that position is the weighted sum of the value rows. A causal mask sets future scores to negative infinity before the softmax. Those positions then get weight zero, not a small weight.

Softmax is applied to each row on its own. One token’s distribution does not constrain another. That independence is what makes the layer parallel across positions. It also means a row can be sharp while its neighbour is flat. The same head can act like a lookup for one token and an average for the next.

The fused version

import torch
import torch.nn.functional as F

q = torch.randn(2, 8, 16, 64)  # batch, heads, tokens, head_dim
k = torch.randn(2, 8, 16, 64)
v = torch.randn(2, 8, 16, 64)

out = F.scaled_dot_product_attention(q, k, v, is_causal=True)
print(out.shape)  # (2, 8, 16, 64)

One query, three keys

Run the attention calculation, then change the query so it points at the second dimension and watch the weights move from “the” to “cat”.

Output
Click "Run" to execute your code

A weighted average, not a lookup

The output is a weighted average of the value rows. Every weight is between zero and one, and the row sums to one. If the distribution is flat, the result is close to the mean of the values. If one score dominates, the result is close to that single value. Most useful attention sits somewhere between those extremes.

Attention therefore cannot copy information that is not in the values. It can select and blend. The content has to be present somewhere in the sequence. A retrieval layer is a different mechanism. Attention is a routing mechanism inside a fixed set of tokens.

The weights are a soft alignment between positions, not a hard pointer. A token can take most of its value from one neighbour and the rest from another. The mix changes with the query. That flexibility lets a head represent a relation that is not exactly one-to-one. A pronoun that depends on two earlier nouns is one example. The head does not have to choose a single source.

The score matrix is n × n per head. Doubling the context quadruples that matrix, which is why long-context models spend so much of their budget on attention memory rather than on parameters.

What the weights do not mean

A high attention weight is tempting to read as an explanation: this token caused that output. The weights are one factor among many. The value vectors, the output projection and the residual stream all change what reaches the next layer. Different heads and different layers produce different patterns for the same input.

Attention maps are still useful. They show where a head routes information. A mask that leaks the future shows up at once as a suspiciously uniform row. Treat them as instrumentation. They are not a story about why the model answered the way it did.

A mask that hides the future

Run the masked softmax for three tokens, then remove the mask and confirm the first row changes because it can now see the last token.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What does the query vector represent?