EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

attention
A weighted blend of value rows, with weights decided by how well each query matches each key.
attention map
The table of scores showing how much each position attended to every other position.
causal mask
A rule that blocks each position from seeing later positions, so predictions can only use the past.
dimension
One axis of a list or grid of numbers; its length is how many values fit along that axis.
distribution
The pattern of how often each possible value appears, or how likely each one is.
dot product
Multiply matching numbers in two lists and add the results; a large total means the lists point the same way.
embedding
A learned list of numbers that stands for a token, with similar tokens placed near each other.
GPU
A processor built for many small calculations at once, which is why matrix work runs well on it.
gradient
A list of slopes, one per number, saying how much each nudge changes the result.
hard pointer
A choice that sends all the weight to one position, instead of spreading it across several.
head
One parallel slice of attention, with its own projections; a layer runs many so it can track different relations at once.
key
What a position advertises it can offer, compared against every query to decide who receives attention.
key axis
The direction in the key vectors whose width decides how far the scores are divided down.
layer
One stage of a model that reads the running vector, transforms it and passes it on.
layer norm
A per-position rescale that keeps the residual stream at a steady size before the next block.
logits
The raw scores a model produces before softmax turns them into probabilities.
matrix
A grid of numbers arranged in rows and columns, used to transform a list into another list.
negative infinity
A value smaller than every real number, used to mark a score that must never be chosen.
one-hot
A vector with a single one and zeros elsewhere, meaning all the weight sits on one choice.
output projection
The learned matrix applied after attention, which turns the mixed values back into the model's width.
parameter
A number inside a model that training adjusts, rather than an input it reads.
positional encoding
Extra numbers added to token embeddings so the model can tell which position each one came from.
projection
A learned matrix that turns a list of numbers into a different list, often shorter.
query
What a position is looking for, compared against every key to decide where to take information from.
residual stream
The running vector each block reads from and adds its output back into.
retrieval layer
A layer whose job is to pull specific stored information into the current position.
saturating
A score so large that the softmax gives almost all the weight to one position.
scale
The step that divides the scores by the square root of the key width, so they stay in a workable range.
score matrix
The grid holding every query's score against every key, one row per position.
soft alignment
Matching by degree rather than by a single choice, so several positions can contribute a little.
softmax
A rule that turns a list of scores into positive weights that sum to one.
token
A chunk of text — a word, part of a word, or a character — that the model treats as one unit.
value
The content a position passes on when attention selects it.
variance
The average squared distance of values from their average, so a spread-out set scores higher.
vector
An ordered list of numbers, where each position is one measured property.
weighted sum
Adding values together with different shares, where the shares come from the attention scores.
weights
The numbers a model multiplies inputs by before adding the results together.

Transformers from the ground up

The feed-forward block

Between attention layers sits a two-layer network applied to each token separately. It holds most of the parameters and does much of the per-token computation.

Part 4 of 10

Advanced15 min readBack to Courses

Position-wise means independent

The feed-forward block uses the same weights at every position. It never looks at another position. Token one and token ten go through the same function separately. That is what “position-wise” means. It is the complement of attention. Attention mixes tokens with each other. The feed-forward block transforms what each token now holds.

The positions are independent. The block is a pair of matrix products over a batch of vectors. Its cost has no term for sequence length. It is therefore cheap next to attention on long inputs. It is fairly expensive on short ones. The two layers are usually the largest source of weights in the model.

The block is applied after attention has mixed the sequence. Each token therefore arrives carrying context from the positions it attended to. The feed-forward network transforms that context without looking sideways. Stacking the two operations lets a model alternate between routing information and working on it. A layer with attention alone can move information around. It can do far less to change what it moved.

Expand, activate, contract

The first matrix projects the model width up to a wider hidden layer. The hidden layer is typically four times as wide. An activation, a simple non-linear rule, is applied element by element. The second matrix projects back to the model width. The output has the same shape as the input. It can therefore be added to the residual stream without any change.

The wider layer is where the model gets room to work. It can represent many more mixes of features than the input width allows. The activation decides which of them fire for this token. The second matrix then sums those values into a single update vector.

The block as two linears

from torch import nn

block = nn.Sequential(
    nn.Linear(512, 2048),
    nn.GELU(),
    nn.Linear(2048, 512),
)

x = block(x)  # (batch, tokens, 512) -> (batch, tokens, 512)

A feed-forward pass

Run the block on a four-dimensional token, then set the hidden width to four and compare the parameter count and the output.

Output
Click "Run" to execute your code

What the block computes

A useful picture is a set of key-value memories. The first matrix compares the token’s vector with many learned directions. The activation decides which directions fire. The second matrix writes the matching output patterns back into the residual stream. The model is not storing facts in a clean database. The analogy still explains why the hidden layer is wide.

Single neurons often respond to features a person can name: a syntax pattern, a language, a topic. They also respond to mixtures, because the model packs far more features than it has neurons. That overlap is called superposition. It explains why removing one neuron can have a small and confusing effect. It does not remove one clean concept.

The width of the hidden layer is a choice about capacity. A narrow layer forces many features through few units. That increases overlap and interference. A wide layer gives each feature more room, at the cost of memory and work. The four-times rule is a starting point. It balances those costs for the model sizes where it was popular. It is not a theoretical optimum.

The four-times expansion is a convention, not a law. A wider hidden layer costs memory in the weights and in the activations; a narrower one saves both and changes what the model can represent. Count the parameters before you copy the default.

GELU and the gated variants

ReLU is the simplest activation. It keeps the positive part and sets the rest to zero. GELU multiplies the input by the chance that a standard normal is below it. That gives a smooth curve and a small negative dip near zero. The smoothness helps training. The negative region lets a unit suppress a signal, not only ignore it.

Modern models often use a gated form. SwiGLU computes two projections. It applies SiLU to one and multiplies it by the other before the output projection. The gate lets the network modulate a value, not only scale it. The cost is a third matrix. The hidden width is usually reduced to keep the weight count similar.

The activation is also a place where serving speed-ups land. ReLU is cheap and sparse. GELU is smoother but more expensive. Gated variants add a matrix but often improve quality per weight. A kernel can fuse the activation into the matrix product. The cost in a real model therefore depends on the code as well as the maths.

Three activations compared

Run the table, then count how many GELU values are negative and explain why a negative activation can still be useful.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What does position-wise mean for the feed-forward block?