Transformers from the ground up

Vocabulary

The terms the chapters assume, in plain words.

attention

A weighted blend of value rows, with weights decided by how well each query matches each key.

attention map

The table of scores showing how much each position attended to every other position.

causal mask

A rule that blocks each position from seeing later positions, so predictions can only use the past.

dimension

One axis of a list or grid of numbers; its length is how many values fit along that axis.

distribution

The pattern of how often each possible value appears, or how likely each one is.

dot product

Multiply matching numbers in two lists and add the results; a large total means the lists point the same way.

embedding

A learned list of numbers that stands for a token, with similar tokens placed near each other.

GPU

A processor built for many small calculations at once, which is why matrix work runs well on it.

gradient

A list of slopes, one per number, saying how much each nudge changes the result.

hard pointer

A choice that sends all the weight to one position, instead of spreading it across several.

head

One parallel slice of attention, with its own projections; a layer runs many so it can track different relations at once.

key

What a position advertises it can offer, compared against every query to decide who receives attention.

key axis

The direction in the key vectors whose width decides how far the scores are divided down.

layer

One stage of a model that reads the running vector, transforms it and passes it on.

layer norm

A per-position rescale that keeps the residual stream at a steady size before the next block.

logits

The raw scores a model produces before softmax turns them into probabilities.

matrix

A grid of numbers arranged in rows and columns, used to transform a list into another list.

negative infinity

A value smaller than every real number, used to mark a score that must never be chosen.

one-hot

A vector with a single one and zeros elsewhere, meaning all the weight sits on one choice.

output projection

The learned matrix applied after attention, which turns the mixed values back into the model's width.

parameter

A number inside a model that training adjusts, rather than an input it reads.

positional encoding

Extra numbers added to token embeddings so the model can tell which position each one came from.

projection

A learned matrix that turns a list of numbers into a different list, often shorter.

query

What a position is looking for, compared against every key to decide where to take information from.

residual stream

The running vector each block reads from and adds its output back into.

retrieval layer

A layer whose job is to pull specific stored information into the current position.

saturating

A score so large that the softmax gives almost all the weight to one position.

scale

The step that divides the scores by the square root of the key width, so they stay in a workable range.

score matrix

The grid holding every query's score against every key, one row per position.

soft alignment

Matching by degree rather than by a single choice, so several positions can contribute a little.

softmax

A rule that turns a list of scores into positive weights that sum to one.

token

A chunk of text — a word, part of a word, or a character — that the model treats as one unit.

value

The content a position passes on when attention selects it.

variance

The average squared distance of values from their average, so a spread-out set scores higher.

vector

An ordered list of numbers, where each position is one measured property.

weighted sum

Adding values together with different shares, where the shares come from the attention scores.

weights

The numbers a model multiplies inputs by before adding the results together.