EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

adapter
A small set of trained numbers that changes how a fixed model behaves and can be added or removed without retraining it.
base model
The original model a tuning run starts from, left unchanged when only an adapter is trained.
catastrophic forgetting
When training on a narrow task makes a model worse at the general skills it already had.
checkpoint
A saved copy of a model's learned numbers at one point in training, kept so a run can be resumed or compared.
dataset
The collection of examples a model learns from, split into training, validation and test parts.
epoch
One full pass over the training examples; several passes let the model see the data more than once.
hyperparameter
A setting chosen before training, such as how fast the model learns, rather than one learned from the data.
leakage
When information from the test or future data reaches training, making the score look better than it is.
learning rate
How far the model's numbers move on each update; too high overshoots and too low barely moves.
LoRA
A method that leaves the model fixed and learns two small grids of numbers whose product is added to chosen parts.
loss
A number that measures how far the model's predictions are from the answers in the data, and that each update tries to lower.
preference tuning
Training a model to choose the better of two answers, rather than to copy one correct answer.
supervised fine-tuning
Training a model on example inputs and the answers you want it to produce for them.
validation set
Examples held back from training and used to choose settings and the stopping point.

Fine-tuning and LoRA

Loss and evaluation

Loss is the number the optimizer minimizes, not the number the product cares about. Evaluate the task, on held-out cases, with the counts behind every rate.

Part 4 of 10

Advanced17 min readBack to Courses

Loss is a proxy

Cross-entropy measures how well the model predicts the next token in your data. It does not measure whether the answer is correct, useful or safe. A model can lower its loss by becoming more confident about the wrong pattern, and a model with a slightly higher loss can be better at the task because it generalises instead of memorising. A lower loss can mean a better fit to the wrong thing. It is a proxy, and every proxy can be gamed by the optimizer.

Use loss to optimise and to detect trouble: a validation loss that turns upward is a clear signal. Use task metrics to decide. If the only number you report is loss, you are reporting how well the model fits the dataset, which is not the question anyone asked. The optimizer is happy to memorize; the product is not. A model that fits the training set perfectly has learned the set, not the task.

Three sets, three jobs

Training fits the weights. Validation chooses the epoch, the learning rate and the early-stopping point. Test is touched once, for the final claim. If you tune on test, it becomes validation and your estimate becomes optimistic. The discipline is boring and it is the only thing that makes the final number mean anything. A test set is a promise that you will not look until the end. Validation is allowed to influence the run; test is not.

Keep the sets fixed across runs. If the test set changes between the base and the tuned model, the comparison mixes the model change with the data change. Version the split with the dataset, and keep the ids so a failure can be traced back to the exact case. A split that cannot be reconstructed is a result that cannot be reproduced. If the split changes, the comparison changes with it, and the old numbers stop meaning anything.

A task metric, not a loss

from evaluate import load

accuracy = load("accuracy")
result = accuracy.compute(
    predictions=["five business days", "two days"],
    references=["five business days", "three days"],
)
print(result["accuracy"])

Pick the epoch by validation loss

Read the four epochs and choose the best checkpoint. Then add a fifth epoch with a lower train loss and a higher validation loss, and explain why it should not ship.

Output
Click "Run" to execute your code

Task metrics and slices

Choose the metric the product can defend. Exact match for a short fact, a rubric or a judge for an open answer, a pass rate for code that either runs or does not. The metric should be computable on every evaluation run and stable enough that a change in it means something. A metric that cannot be computed on a single case is hard to debug. Choose it before the run and write it down, or the metric will be chosen after the result.

Report slices, not only the average. A model can improve overall while getting worse on long inputs, one language, or the intent that matters most. Slice by the dimensions in the data: length, class, tenant, source. The slice should be chosen before the run, not after the failures appear. A slice with ten cases is a hint, not a verdict. Report the count for every slice, or a one-case difference will look like a trend.

Compare base and tuned on the same cases, the same prompt and the same decoding settings. A win that comes from a different temperature is not a win.

Small n and significance

Twenty cases cannot distinguish eighty percent from ninety percent. The interval around a small proportion is wide, and a two-case difference is noise. Report the count and an interval, or use a paired comparison: for each case, did the tuned model win, lose or tie? A confidence interval makes the uncertainty visible; a bare percentage hides it behind a decimal point. If the interval overlaps the baseline, the honest conclusion is that the run did not settle the question.

A paired result of twelve wins and two losses is promising; seven and five is not evidence of anything. The paired view also tells you where the wins came from, which is more useful than a single accuracy number. If the wins are all in one slice, the headline is a slice result wearing an average. A paired comparison also gives you the cases to read, which is often more useful than the count. Read the losing cases before the winning ones; they are where the next dataset change comes from.

Compare base and tuned case by case

Count wins, losses and ties, then break the result down by slice. Add a slice where the tuned model loses and decide whether the overall win is still convincing.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

Why is validation loss not enough to decide whether a model is better?