EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

attention
A weighted blend of value rows, with weights decided by how well each query matches each key.
attention map
The table of scores showing how much each position attended to every other position.
causal mask
A rule that blocks each position from seeing later positions, so predictions can only use the past.
dimension
One axis of a list or grid of numbers; its length is how many values fit along that axis.
distribution
The pattern of how often each possible value appears, or how likely each one is.
dot product
Multiply matching numbers in two lists and add the results; a large total means the lists point the same way.
embedding
A learned list of numbers that stands for a token, with similar tokens placed near each other.
GPU
A processor built for many small calculations at once, which is why matrix work runs well on it.
gradient
A list of slopes, one per number, saying how much each nudge changes the result.
hard pointer
A choice that sends all the weight to one position, instead of spreading it across several.
head
One parallel slice of attention, with its own projections; a layer runs many so it can track different relations at once.
key
What a position advertises it can offer, compared against every query to decide who receives attention.
key axis
The direction in the key vectors whose width decides how far the scores are divided down.
layer
One stage of a model that reads the running vector, transforms it and passes it on.
layer norm
A per-position rescale that keeps the residual stream at a steady size before the next block.
logits
The raw scores a model produces before softmax turns them into probabilities.
matrix
A grid of numbers arranged in rows and columns, used to transform a list into another list.
negative infinity
A value smaller than every real number, used to mark a score that must never be chosen.
one-hot
A vector with a single one and zeros elsewhere, meaning all the weight sits on one choice.
output projection
The learned matrix applied after attention, which turns the mixed values back into the model's width.
parameter
A number inside a model that training adjusts, rather than an input it reads.
positional encoding
Extra numbers added to token embeddings so the model can tell which position each one came from.
projection
A learned matrix that turns a list of numbers into a different list, often shorter.
query
What a position is looking for, compared against every key to decide where to take information from.
residual stream
The running vector each block reads from and adds its output back into.
retrieval layer
A layer whose job is to pull specific stored information into the current position.
saturating
A score so large that the softmax gives almost all the weight to one position.
scale
The step that divides the scores by the square root of the key width, so they stay in a workable range.
score matrix
The grid holding every query's score against every key, one row per position.
soft alignment
Matching by degree rather than by a single choice, so several positions can contribute a little.
softmax
A rule that turns a list of scores into positive weights that sum to one.
token
A chunk of text — a word, part of a word, or a character — that the model treats as one unit.
value
The content a position passes on when attention selects it.
variance
The average squared distance of values from their average, so a spread-out set scores higher.
vector
An ordered list of numbers, where each position is one measured property.
weighted sum
Adding values together with different shares, where the shares come from the attention scores.
weights
The numbers a model multiplies inputs by before adding the results together.

Transformers from the ground up

Residual streams and layer norm

Every sublayer writes into a shared residual stream. Layer norm keeps that stream at a workable scale, and the order of the two operations decides how deep a model can be trained.

Part 5 of 10

Advanced16 min readBack to Courses

The residual stream

Each block reads the current vector for a token. It computes an update and adds that update back to the vector it read. The path from input to output is therefore a sum of many small contributions. It is not a chain of replacements. The identity connection gives gradients a short route to every earlier layer. That is what makes deep stacks trainable.

The image to keep is a shared bus. Attention and the feed-forward block both read from it and write onto it. Neither owns it. Because the stream is a sum, a later layer can still recover the original token signal. Many blocks may have added to it by then. The stream’s scale also tends to grow with depth unless something controls it.

Layer norm, not batch norm

Layer norm rescales each token’s vector across its features. It subtracts the mean and divides by the standard deviation. It then applies a learned scale and shift. Every token is normalised using only its own values. The result does not depend on the other examples in the batch or on the sequence length. That independence is why it works for text of any length and for small batches.

Batch norm, by contrast, computes statistics across the batch. It works well for images with a fixed shape and a large batch. It also couples examples together and behaves differently in training and inference. Layer norm has no running averages to maintain. Its behaviour is the same in both modes. The learned scale and shift restore the network’s ability to represent unusual distributions.

Without those two vectors, every token would sit on a sphere of the same radius. The next matrix would have to work with directions alone. The scale lets a feature grow again. The shift lets it sit away from zero. Normalisation is therefore a change of variables, not a constraint. The cost is small: two vectors per layer, not a full matrix.

A pre-norm block

from torch import nn

norm = nn.LayerNorm(512, eps=1e-5)

# normalize, transform, then add to the stream
x = x + attention(norm(x))
x = x + feed_forward(norm(x))

Normalise one token

Run the normalisation, then change the learned scale to [1, 2, 3, 4] and confirm the output mean is no longer zero.

Output
Click "Run" to execute your code

Pre-norm and post-norm

The original transformer normalised after the residual addition: x = norm(x + sublayer(x)). That post-norm arrangement needs a careful learning-rate warmup. Very deep stacks also become hard to train, because the residual path is rescaled at every block. Many later models moved the norm inside the branch, before the sublayer. They kept a final norm before the output.

Pre-norm is easier to train and is the common choice for deep models. It is not strictly better in every respect. The residual stream is never normalised between blocks, so its scale can grow with depth. Some work also finds that post-norm or hybrid arrangements give better quality per weight at shallow depths. The choice is empirical, not a law.

The placement also changes what the residual stream carries. In pre-norm, the stream is a running sum of unnormalised updates. Its norm can grow with depth. In post-norm, every block resets the scale before the next one reads it. Neither is free. One asks for a final norm and careful starting weights. The other asks for a warmup schedule.

Layer norm normalises across features, not across tokens. If you accidentally normalise the other axis, every token in a sequence becomes coupled and the model stops behaving like a language model at all.

What normalization does to the stream

Normalisation removes the overall scale of a vector and keeps its direction. The next matrix product is then less sensitive to how large the stream has become. The activation also stays in a range where gradients are useful. The learned scale can put some of the size back. It does so per feature, not per token.

RMSNorm is a simpler version. It skips the mean subtraction and divides by the root mean square. It is cheaper and works about as well in many models. The epsilon term matters when the variance is near zero. Without it, a constant vector would divide by zero and produce an infinite update. The value is small, but it is not decorative.

A final layer norm before the output head is common in pre-norm models. The residual stream is never normalised inside the blocks. Without the final norm, the logits scale with the depth and the softmax can saturate. The final norm is cheap and easy to forget. Its absence shows up as unstable generation, not as a shape error.

A stream that survives depth

Run five blocks with and without the residual addition, then make the block shrink its input and compare the two norms again.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What does a residual connection add?