EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

adapter
A small set of trained numbers that changes how a fixed model behaves and can be added or removed without retraining it.
base model
The original model a tuning run starts from, left unchanged when only an adapter is trained.
catastrophic forgetting
When training on a narrow task makes a model worse at the general skills it already had.
checkpoint
A saved copy of a model's learned numbers at one point in training, kept so a run can be resumed or compared.
dataset
The collection of examples a model learns from, split into training, validation and test parts.
epoch
One full pass over the training examples; several passes let the model see the data more than once.
hyperparameter
A setting chosen before training, such as how fast the model learns, rather than one learned from the data.
leakage
When information from the test or future data reaches training, making the score look better than it is.
learning rate
How far the model's numbers move on each update; too high overshoots and too low barely moves.
LoRA
A method that leaves the model fixed and learns two small grids of numbers whose product is added to chosen parts.
loss
A number that measures how far the model's predictions are from the answers in the data, and that each update tries to lower.
preference tuning
Training a model to choose the better of two answers, rather than to copy one correct answer.
supervised fine-tuning
Training a model on example inputs and the answers you want it to produce for them.
validation set
Examples held back from training and used to choose settings and the stopping point.

Fine-tuning and LoRA

Catastrophic forgetting

A narrow fine-tune can make a model better at one task and worse at everything else. Measure the general suite, mix replay data, and keep a rollback.

Part 7 of 10

Advanced16 min readBack to Courses

What forgetting looks like

The tuned model gets better at the narrow task and worse at everything else. Instruction following slips, refusals become inconsistent, formatting drifts, and a general question that the base handled confidently now gets a hesitant or oddly shaped answer. It is not memory loss in the human sense; the weights have moved toward your distribution and away from the one they were pretrained on. The model does not forget how to speak; it forgets which register the product needs. It can also get better at the task and worse at saying no.

The effect is easy to miss because the target metric improves. A support classifier that gains four points on refunds can lose ten points on an unrelated intent that nobody thought to test. The only way to see it is to evaluate a general suite before and after, on the same cases, with the same prompt. The loss has no term for the behaviour you did not include in the data. Full fine-tuning moves every weight, so every behaviour is in scope for the update.

Why it happens

Narrow data, many epochs, a high learning rate and full fine-tuning all push in the same direction. The gradient only carries a signal about your examples; nothing in it says “preserve the rest”. With enough steps, the model can fit your task by overwriting circuits that handled other tasks, especially when the dataset is small and repetitive. A general suite is a smoke test for the rest of the product, not a benchmark competition. Run it on the base model first, so the baseline is not reconstructed from memory.

LoRA constrains the update to a low-rank subspace, which reduces the damage but does not eliminate it. A rank that is high enough to learn the task is also high enough to interfere. The mitigation is not a single setting; it is a combination of a smaller update, fewer epochs, replay data and an evaluation that notices the loss. Replay is cheaper than retraining from scratch and more reliable than hoping. A replay set that is too small is a token gesture; one that is too large hides the task.

Mix the task data with replay data

from datasets import interleave_datasets

mixed = interleave_datasets(
    [task_dataset, general_dataset],
    probabilities=[0.85, 0.15],
    seed=42,
)
print(len(mixed), mixed[0]["messages"][0]["role"])

A general suite catches the trade

Compare the base and tuned scores across four suites. Then add a suite that drops by twelve points and decide whether the task gain still justifies shipping.

Output
Click "Run" to execute your code

Mix replay data

Replay means keeping a share of general instruction data in the training mixture. Five to twenty percent is a common starting range. The model sees the task and the general distribution in the same run, so the update has a reason to preserve the behaviour that the general examples exercise. The exact share is a hyperparameter, not a law. The mix ratio is a dial between learning the task and keeping the rest. The share should be recorded with the run, because it changes the meaning of the result.

Too much replay slows task learning and can dilute the behaviour you wanted. Too little and the general suite drops. Match the replay data to the prompts production actually sees: if the product uses tools and long context, replaying only short trivia questions will not protect the tool behaviour. Measure the share and the result, and write both next to the run. A small amount of replay can protect a lot of behaviour, which is why it is the first mitigation to try. LoRA reduces the blast radius but does not make the update free.

Evaluate general ability with the same harness before and after. A task win with a ten-point general drop is a bad trade for a product that does more than one thing.

Mitigations that work

Use LoRA instead of full fine-tuning, lower the learning rate, train for fewer epochs, and stop on validation. Add replay data, keep weight decay, and merge the adapter at full or reduced scale depending on the regression. None of these is a guarantee; together they turn a risky update into a controlled one. A lower learning rate and fewer epochs are the same idea applied to the update size. Keep the previous adapter, not only the previous base.

Keep a rollback that is not “retrain”. The base model id plus the previous adapter version is a rollback you can deploy in minutes. If the general suite drops after release, you want a switch, not an investigation that starts with a training run. The rollback should be tested before the launch, not discovered during the incident.

How much replay do you need?

Compute the replay examples needed for a fifteen percent general share, then check whether the available pool is large enough. Try a twenty-five percent target.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What does catastrophic forgetting look like after a narrow fine-tune?