EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

attention
A weighted blend of value rows, with weights decided by how well each query matches each key.
attention map
The table of scores showing how much each position attended to every other position.
causal mask
A rule that blocks each position from seeing later positions, so predictions can only use the past.
dimension
One axis of a list or grid of numbers; its length is how many values fit along that axis.
distribution
The pattern of how often each possible value appears, or how likely each one is.
dot product
Multiply matching numbers in two lists and add the results; a large total means the lists point the same way.
embedding
A learned list of numbers that stands for a token, with similar tokens placed near each other.
GPU
A processor built for many small calculations at once, which is why matrix work runs well on it.
gradient
A list of slopes, one per number, saying how much each nudge changes the result.
hard pointer
A choice that sends all the weight to one position, instead of spreading it across several.
head
One parallel slice of attention, with its own projections; a layer runs many so it can track different relations at once.
key
What a position advertises it can offer, compared against every query to decide who receives attention.
key axis
The direction in the key vectors whose width decides how far the scores are divided down.
layer
One stage of a model that reads the running vector, transforms it and passes it on.
layer norm
A per-position rescale that keeps the residual stream at a steady size before the next block.
logits
The raw scores a model produces before softmax turns them into probabilities.
matrix
A grid of numbers arranged in rows and columns, used to transform a list into another list.
negative infinity
A value smaller than every real number, used to mark a score that must never be chosen.
one-hot
A vector with a single one and zeros elsewhere, meaning all the weight sits on one choice.
output projection
The learned matrix applied after attention, which turns the mixed values back into the model's width.
parameter
A number inside a model that training adjusts, rather than an input it reads.
positional encoding
Extra numbers added to token embeddings so the model can tell which position each one came from.
projection
A learned matrix that turns a list of numbers into a different list, often shorter.
query
What a position is looking for, compared against every key to decide where to take information from.
residual stream
The running vector each block reads from and adds its output back into.
retrieval layer
A layer whose job is to pull specific stored information into the current position.
saturating
A score so large that the softmax gives almost all the weight to one position.
scale
The step that divides the scores by the square root of the key width, so they stay in a workable range.
score matrix
The grid holding every query's score against every key, one row per position.
soft alignment
Matching by degree rather than by a single choice, so several positions can contribute a little.
softmax
A rule that turns a list of scores into positive weights that sum to one.
token
A chunk of text — a word, part of a word, or a character — that the model treats as one unit.
value
The content a position passes on when attention selects it.
variance
The average squared distance of values from their average, so a spread-out set scores higher.
vector
An ordered list of numbers, where each position is one measured property.
weighted sum
Adding values together with different shares, where the shares come from the attention scores.
weights
The numbers a model multiplies inputs by before adding the results together.

Transformers from the ground up

The training objective

A language model is trained to predict the next token. One shifted copy of the sequence supplies every target, and cross-entropy turns logits into a loss.

Part 7 of 10

Advanced16 min readBack to Courses

One prediction per position

A causal language model reads tokens from left to right and predicts the next one. At position t it sees tokens up to t. It then produces a distribution over the vocabulary for token t + 1. The same forward pass gives a prediction at every position. A single sequence of length n therefore supplies n − 1 training examples, not one.

The targets are the inputs shifted by one. Take the sequence “the cat sat on the mat”. The first position learns to predict “cat” from “the”. The second learns to predict “sat” from “the cat”, and so on. This is teacher forcing. The model is given the true prefix at every step, not the token it would have generated itself.

The causal mask is what makes this objective honest. Without it, position t could look at token t + 1. The task would then collapse into copying the target. The mask is applied inside every attention layer, not once at the input. A later layer could otherwise route future information back through an earlier position. A single leak anywhere in the stack is enough to make the loss meaningless.

Logits, softmax, cross-entropy

The final layer produces one score per vocabulary entry for each position. Those scores are logits, not probabilities. A softmax turns a row into a distribution. Cross-entropy takes the negative log of the probability given to the correct token. A confident correct prediction costs almost nothing. A confident wrong one costs a lot.

The loss for a batch is the average over all predicted positions. Perplexity is the exponential of that average. It puts the loss back on a readable scale. Think of it as the number of equally likely options the model is choosing between. A perplexity of 20 means the model is about as unsure as a uniform choice among twenty tokens.

The shifted loss in PyTorch

from torch import nn

loss_fn = nn.CrossEntropyLoss(ignore_index=-100)
logits = model(input_ids).logits  # (batch, tokens, vocab)

loss = loss_fn(
    logits[:, :-1].reshape(-1, vocab),
    input_ids[:, 1:].reshape(-1),
)
print(loss.item())

Cross-entropy by hand

Run the three predictions, then change the first logit row so “the” is the top token and watch the mean loss and perplexity rise.

Output
Click "Run" to execute your code

What the loss does not say

A low loss means the model gives high probability to the next token in the training data. It does not mean the sentence is true, useful or safe. The objective rewards imitation of the corpus, including its errors, its repetitions and its biases. Every later alignment step is a correction on top of an objective that was never about truth.

Teacher forcing also creates a mismatch. During training the model always sees a correct prefix. During generation it sees its own output, mistakes included. A small error can push the model into a state it never saw in training. The rest of the sequence is then generated from there. Exposure bias is one reason a model can look fluent under teacher forcing and drift when it writes on its own.

The loss is also computed per token. A long document therefore contributes many more terms than a short one. That is usually what you want for language modelling. It also means the average is dominated by the noisiest and most repetitive parts of the corpus. Weighting or filtering the data changes the objective as much as changing the model does.

Do not compare perplexity across tokenizers. A finer tokenizer makes each prediction easier, so the same model can look better only because the vocabulary changed. Compare loss only with the same tokenizer and the same evaluation text.

Masking and packing

Padding tokens must not contribute to the loss. The usual convention is to set their targets to a sentinel value such as −100. The loss function is then told to ignore that value. Without the mask, the model is trained to predict padding. The average loss is also diluted by positions that carry no information.

Packing several short documents into one sequence improves hardware use. It also changes what the model can see. Without a block-diagonal attention mask, the end of one document can attend to the start of the next. That is a quiet form of leakage. It can teach the model to continue across boundaries that should be independent. The mask and the position ids both have to respect the boundary.

Packing also changes the effective batch. A thousand short examples can become a handful of long sequences. That improves throughput and changes the gradient. If the documents are unrelated, the model may learn transitions that never occur in reality. The boundary mask is not a detail of speed. It is part of the definition of the task.

Shifted targets and a bigram model

Run the shifted pairs and the smoothed bigram loss, then add a second sentence and see which transitions change.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What target does a causal language model predict at each position?