EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

attention
A weighted blend of value rows, with weights decided by how well each query matches each key.
attention map
The table of scores showing how much each position attended to every other position.
causal mask
A rule that blocks each position from seeing later positions, so predictions can only use the past.
dimension
One axis of a list or grid of numbers; its length is how many values fit along that axis.
distribution
The pattern of how often each possible value appears, or how likely each one is.
dot product
Multiply matching numbers in two lists and add the results; a large total means the lists point the same way.
embedding
A learned list of numbers that stands for a token, with similar tokens placed near each other.
GPU
A processor built for many small calculations at once, which is why matrix work runs well on it.
gradient
A list of slopes, one per number, saying how much each nudge changes the result.
hard pointer
A choice that sends all the weight to one position, instead of spreading it across several.
head
One parallel slice of attention, with its own projections; a layer runs many so it can track different relations at once.
key
What a position advertises it can offer, compared against every query to decide who receives attention.
key axis
The direction in the key vectors whose width decides how far the scores are divided down.
layer
One stage of a model that reads the running vector, transforms it and passes it on.
layer norm
A per-position rescale that keeps the residual stream at a steady size before the next block.
logits
The raw scores a model produces before softmax turns them into probabilities.
matrix
A grid of numbers arranged in rows and columns, used to transform a list into another list.
negative infinity
A value smaller than every real number, used to mark a score that must never be chosen.
one-hot
A vector with a single one and zeros elsewhere, meaning all the weight sits on one choice.
output projection
The learned matrix applied after attention, which turns the mixed values back into the model's width.
parameter
A number inside a model that training adjusts, rather than an input it reads.
positional encoding
Extra numbers added to token embeddings so the model can tell which position each one came from.
projection
A learned matrix that turns a list of numbers into a different list, often shorter.
query
What a position is looking for, compared against every key to decide where to take information from.
residual stream
The running vector each block reads from and adds its output back into.
retrieval layer
A layer whose job is to pull specific stored information into the current position.
saturating
A score so large that the softmax gives almost all the weight to one position.
scale
The step that divides the scores by the square root of the key width, so they stay in a workable range.
score matrix
The grid holding every query's score against every key, one row per position.
soft alignment
Matching by degree rather than by a single choice, so several positions can contribute a little.
softmax
A rule that turns a list of scores into positive weights that sum to one.
token
A chunk of text — a word, part of a word, or a character — that the model treats as one unit.
value
The content a position passes on when attention selects it.
variance
The average squared distance of values from their average, so a spread-out set scores higher.
vector
An ordered list of numbers, where each position is one measured property.
weighted sum
Adding values together with different shares, where the shares come from the attention scores.
weights
The numbers a model multiplies inputs by before adding the results together.

Transformers from the ground up

Tokenisation in depth

A tokenizer is a learned merge table, not a rule about words. Read the merges, measure fertility, and see why a vocabulary change is a model change.

Part 8 of 10

Advanced16 min readBack to Courses

Merges, not rules

Byte-pair encoding learns a vocabulary instead of being handed one. It starts from a table of single characters. It counts every adjacent pair in a corpus and merges the most frequent pair into a new symbol. It repeats until the vocabulary reaches its target size. The result is an ordered merge table. That table is the tokenizer.

Order matters, because the rules are applied in the order they were learned. A later merge can only combine symbols that earlier merges produced. Two tokenizers with the same vocabulary size can therefore disagree on almost every id. The corpus, the split into words and the merge order all differ. Reproducing a model means shipping the merge table with it.

The frequency that drives a merge is counted over words, weighted by how often they appear. It is not counted over distinct words. A pair that appears once in ten thousand rare words matters less than a pair inside “the”. Code keeps a dictionary from word to count. The pair count runs over that dictionary. That is what makes training on billions of tokens tractable.

Bytes, so nothing is unknown

A character vocabulary cannot cover every script. A word vocabulary cannot cover a new word. Byte-level BPE avoids both problems by starting from the 256 possible bytes. Every UTF-8 string is a sequence of bytes. Every string can therefore be encoded, and there is no unknown token at all.

The cost moves rather than disappears. A character outside the languages the corpus favoured takes several bytes. So it takes several tokens. The same sentence in English and in a rare script can take three times as many tokens. That triples the cost, and the share of the context window it fills.

The printable mapping is a detail with consequences. Bytes are mapped to visible characters so the tokenizer file stays text. So a token can look like “Ġthe”, a space marker followed by letters. Reading the raw token list is useful. It is not the same as reading the original string.

The same merges in a library

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("tokenizer.json")
encoded = tokenizer.encode("unhappiness costs three tokens")

print(encoded.tokens)          # the pieces, including markers
print(encoded.ids)             # what the model reads
print(encoded.offsets)         # where each piece came from
print(tokenizer.decode(encoded.ids))

Learn the merges

Run five merge steps over the four weighted words, then add a fifth word to the corpus and see which merge changes first.

Output
Click "Run" to execute your code

Fertility and the vocabulary budget

Vocabulary size trades sequence length against row quality. A larger vocabulary turns common words into one token. That shortens sequences and cuts attention cost. It also gives each token a rarer row to learn and widens the output softmax. A smaller vocabulary keeps rows well trained. It makes every sentence longer.

Fertility is the average number of tokens per word. English prose often lands near 1.3 with a modern vocabulary. Code, numbers and other scripts run higher. Measure it on your own data rather than trusting a published figure. The tokenizer that matters is the one attached to the model you are using.

Compression ratio is characters per token. It is the same measurement inverted, and it is the one to log in a pipeline. A sudden drop after a data change means the new data is being shredded. The first symptom is usually a bill or a truncated prompt, not an error message.

Changing the tokenizer invalidates the embedding table. The ids no longer point at the same rows, so a checkpoint cannot be reused with a new vocabulary without retraining at least the embeddings.

Normalisation and special tokens

Before any merge, the text is normalised, or cleaned up, and split. Normalisation may lower-case, strip accents or collapse whitespace. Pre-tokenisation, the split into words, cuts on spaces, punctuation and digit boundaries. A merge then never crosses a boundary it should respect. Both steps are part of the tokenizer, and both change the ids.

Special tokens occupy ids in the same vocabulary. A beginning-of-sequence marker, a padding token, a mask token and chat role markers are all rows. The model can read them and predict them unless they are frozen. The attention mask must exclude padding, and the loss must ignore it. Otherwise the model learns to produce padding as if it were text.

The split between normalisation and pre-tokenisation also decides what the model can never see. A tokenizer that strips accents cannot tell apart two words that differ only by one. A tokenizer that lower-cases loses case at the first step. Those choices are cheap to make and expensive to reverse.

Encode with the ranks

Encode two sentences with the merge table, then reorder two merges and see which words are split differently and how fertility moves.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What is a trained byte-pair encoding tokenizer?