EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

attention
A weighted blend of value rows, with weights decided by how well each query matches each key.
attention map
The table of scores showing how much each position attended to every other position.
causal mask
A rule that blocks each position from seeing later positions, so predictions can only use the past.
dimension
One axis of a list or grid of numbers; its length is how many values fit along that axis.
distribution
The pattern of how often each possible value appears, or how likely each one is.
dot product
Multiply matching numbers in two lists and add the results; a large total means the lists point the same way.
embedding
A learned list of numbers that stands for a token, with similar tokens placed near each other.
GPU
A processor built for many small calculations at once, which is why matrix work runs well on it.
gradient
A list of slopes, one per number, saying how much each nudge changes the result.
hard pointer
A choice that sends all the weight to one position, instead of spreading it across several.
head
One parallel slice of attention, with its own projections; a layer runs many so it can track different relations at once.
key
What a position advertises it can offer, compared against every query to decide who receives attention.
key axis
The direction in the key vectors whose width decides how far the scores are divided down.
layer
One stage of a model that reads the running vector, transforms it and passes it on.
layer norm
A per-position rescale that keeps the residual stream at a steady size before the next block.
logits
The raw scores a model produces before softmax turns them into probabilities.
matrix
A grid of numbers arranged in rows and columns, used to transform a list into another list.
negative infinity
A value smaller than every real number, used to mark a score that must never be chosen.
one-hot
A vector with a single one and zeros elsewhere, meaning all the weight sits on one choice.
output projection
The learned matrix applied after attention, which turns the mixed values back into the model's width.
parameter
A number inside a model that training adjusts, rather than an input it reads.
positional encoding
Extra numbers added to token embeddings so the model can tell which position each one came from.
projection
A learned matrix that turns a list of numbers into a different list, often shorter.
query
What a position is looking for, compared against every key to decide where to take information from.
residual stream
The running vector each block reads from and adds its output back into.
retrieval layer
A layer whose job is to pull specific stored information into the current position.
saturating
A score so large that the softmax gives almost all the weight to one position.
scale
The step that divides the scores by the square root of the key width, so they stay in a workable range.
score matrix
The grid holding every query's score against every key, one row per position.
soft alignment
Matching by degree rather than by a single choice, so several positions can contribute a little.
softmax
A rule that turns a list of scores into positive weights that sum to one.
token
A chunk of text — a word, part of a word, or a character — that the model treats as one unit.
value
The content a position passes on when attention selects it.
variance
The average squared distance of values from their average, so a spread-out set scores higher.
vector
An ordered list of numbers, where each position is one measured property.
weighted sum
Adding values together with different shares, where the shares come from the attention scores.
weights
The numbers a model multiplies inputs by before adding the results together.

Transformers from the ground up

Tokens and embeddings

A transformer never sees letters. It sees integer ids from a fixed vocabulary, mapped to vectors that move during training. Tokenisation is the first place a model can fail.

Part 1 of 10

Intermediate16 min readBack to Courses

The model sees integers

Text arrives as a string and leaves the tokenizer as a list of integers. A tokenizer splits the string into pieces. A piece can be a whole word, a word fragment, a punctuation mark or a single byte. The tokenizer looks each piece up in a vocabulary, a fixed list of pieces set before training. The model never sees the original characters. It sees the ids, plus whatever position signal the model adds.

Modern tokenizers learn their pieces with byte-pair encoding. Start from characters. Count adjacent pairs in a large corpus. Merge the most frequent pair, and repeat until the vocabulary reaches its target size. A frequent word survives as one token. A rare word splits into the pieces that built it. So “unhappiness” can cost three tokens while “the” costs one.

A vocabulary is usually between 30,000 and 200,000 entries. The exact size is a trade. A larger vocabulary makes common text shorter. It also gives every token a rarer row to learn, and it makes the output softmax wider. A tokenizer trained on English prose will still spend several tokens on a chemical formula or a line of Chinese. Those patterns were not frequent enough to earn a merge.

An embedding is a lookup table

The first layer is a matrix with one row per vocabulary entry and one column per model dimension. An embedding is a lookup: token id 4,271 selects row 4,271. That row is the token’s starting vector. Nothing is computed at this step, which is why the layer is cheap. Its parameters are still learned. Gradients from every occurrence of the token push the same row.

Tokens used in similar contexts end up with vectors that point in similar directions. Cosine similarity is the usual measure. It compares the angle between two rows and ignores their length. “Cat” and “dog” are close. “Cat” and “the” are not. The geometry summarises how the words were used in the corpus. It is not a definition, and it carries the biases of that corpus.

The same two steps in a library

from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")

batch = tokenizer(["a cat sat", "on the mat"], return_tensors="pt", padding=True)
output = model(**batch)
print(batch["input_ids"].shape)       # (2, tokens)
print(output.last_hidden_state.shape) # (2, tokens, 768)

A tokenizer you can read

Run the greedy tokenizer on four words, then add a merged piece to the vocabulary and tokenise “cats” again.

Output
Click "Run" to execute your code

Tokenisation leaks into behaviour

The vocabulary is fixed, so it decides what the model can see cheaply. Numbers are a familiar failure. A tokenizer trained mostly on prose may split “2024” into two or three pieces. Arithmetic on years is then harder than arithmetic on small integers. The same text in a language rare in the corpus often costs two or three times as many tokens as English.

That cost is not cosmetic. The context window is measured in tokens. So is the bill. Attention time grows with the square of the sequence length. A prompt that looks short in words can be long in ids. Counting words is a rough proxy at best. Counting tokens is the measurement that matters.

Special tokens are part of the same vocabulary and are easy to forget. A beginning-of-sequence marker, a padding token and a mask token all occupy ids. The model can read them and predict them. If a padding token is not masked out of the loss, the model learns to produce it. A generation that ends with it then looks like a bug in the sampler. The real bug is in the training data.

Log the token count next to the character count when you build a prompt. A sudden jump usually means a rare word, a number or a code block is being shredded into single characters.

From ids to vectors, and back

At the output end, the final hidden state for each position is multiplied by a matrix. That matrix maps the vector back to vocabulary size. The scores are logits, raw numbers before any normalisation. A softmax turns them into a distribution over the next token. Many models tie the output matrix to the input embedding matrix. The same rows are then read at the start and written at the end.

Tying saves parameters and gives the geometry a second job. A token’s input direction is also the direction the model must produce to predict it. The vector for a token is therefore shaped by two things. One is what it means when read. The other is what it must look like when chosen. That shared pressure is one reason the space is more structured than a random lookup table.

Vectors that can do arithmetic

Run the analogy, then add vectors for paris, france and italy and check whether the same subtraction and addition lands on the right city.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What does a tokenizer hand to the model?