EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

adapter
A small set of trained numbers that changes how a fixed model behaves and can be added or removed without retraining it.
base model
The original model a tuning run starts from, left unchanged when only an adapter is trained.
catastrophic forgetting
When training on a narrow task makes a model worse at the general skills it already had.
checkpoint
A saved copy of a model's learned numbers at one point in training, kept so a run can be resumed or compared.
dataset
The collection of examples a model learns from, split into training, validation and test parts.
epoch
One full pass over the training examples; several passes let the model see the data more than once.
hyperparameter
A setting chosen before training, such as how fast the model learns, rather than one learned from the data.
leakage
When information from the test or future data reaches training, making the score look better than it is.
learning rate
How far the model's numbers move on each update; too high overshoots and too low barely moves.
LoRA
A method that leaves the model fixed and learns two small grids of numbers whose product is added to chosen parts.
loss
A number that measures how far the model's predictions are from the answers in the data, and that each update tries to lower.
preference tuning
Training a model to choose the better of two answers, rather than to copy one correct answer.
supervised fine-tuning
Training a model on example inputs and the answers you want it to produce for them.
validation set
Examples held back from training and used to choose settings and the stopping point.

Fine-tuning and LoRA

Hyperparameters that matter

Learning rate, epochs and effective batch decide most of the result. Rank and alpha tune capacity; everything else is a smaller correction.

Part 6 of 10

Advanced16 min readBack to Courses

Learning rate comes first

The learning rate decides how far each update moves the weights. LoRA usually tolerates something in the range of 1e-4 to 3e-4; full fine-tuning is an order of magnitude smaller, around 1e-5 to 2e-5, because every weight is moving. Too high and the loss spikes or the model forgets; too low and it never reaches the behaviour in the time you have. The peak rate is the number to remember; the schedule is a shape around it. The first hundred steps tell you whether the rate is in the right range.

Warm up over the first few percent of steps, then decay. Warmup protects the pretrained weights from a large first update, and a cosine or linear decay lets the model settle. The schedule matters less than the peak, but a missing warmup is a common reason a run looks broken in the first twenty steps. A warmup that is too long wastes the first part of a short run. Cosine decay is a default, not a requirement; a linear schedule is easier to reason about.

Epochs and the effective batch

Supervised fine-tuning usually wants one to three epochs. More than that and a small dataset is memorised; the training loss keeps falling while the validation loss turns up. If the task is genuinely large and diverse, more epochs can help, but the validation curve should be the reason, not a default. Epochs are a count of passes, not a measure of learning. One epoch on a large, diverse set can be more than three on a small one.

The effective batch is the per-device batch times the gradient accumulation steps times the number of devices. A larger batch means fewer optimizer steps for the same number of examples, so the learning rate may need to rise with it. Sequence length is the other multiplier: attention memory grows faster than linearly, and a long sequence can turn a comfortable batch into an out-of-memory error. A batch that is too small makes the gradient noisy; one that is too large makes each step expensive. A long sequence also changes the effective batch, because padding wastes memory.

The arguments that carry the recipe

from transformers import TrainingArguments

args = TrainingArguments(
    output_dir="out",
    learning_rate=2e-4,
    num_train_epochs=2,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    warmup_ratio=0.03,
    lr_scheduler_type="cosine",
    bf16=True,
    gradient_checkpointing=True,
    report_to="none",
)

Effective batch and total steps

Compute the effective batch, steps per epoch and tokens seen for three configs. Then change the accumulation on the smallest config until its effective batch matches the largest.

Output
Click "Run" to execute your code

Rank, alpha and dropout

Rank is the adapter’s capacity, alpha is the scale of its update, and dropout is a small regulariser on the LoRA path. Increasing rank without more data tends to overfit; increasing alpha without changing rank makes each update larger, which can look like a higher learning rate. The ratio of alpha to rank is the number to keep stable when you compare runs. Rank and alpha are meaningless without the data size they are fitted to. A rank sweep with one seed is a coin toss.

Tune rank after the learning rate and the epoch count, because those two dominate. Keep a seed and run at least two seeds on a small task before believing a one-point difference. A grid search on a noisy metric finds noise, and a configuration that wins by luck will not survive the next dataset. Rank is the last thing to tune, because it interacts with the amount of data. A run with a good rank and a bad rate is still a bad run.

Change one hyperparameter at a time and keep the evaluation fixed. If two things change together, the result tells you that something happened, not what caused it.

Recipes and budgets

Start from a recipe known to work for the base model, then adapt it to your data. Log the full configuration with the run: base model, rank, alpha, targets, learning rate, batch, epochs, sequence length, seed and dataset version. A run without its config is an anecdote, even if the adapter file survives. A seed is a hyperparameter you do not tune but must record. A config that lives only in a terminal scroll is not a record.

Estimate the compute before you start: tokens seen is examples times average tokens times epochs, and GPU-hours is steps times step time divided by 3600. Add the evaluation runs and a retry or two. Early stopping on validation is the cheapest way to avoid paying for the epochs after the model has already peaked. The cheapest experiment is often the one that changes nothing but the seed.

A small grid, read honestly

Sort the runs by validation loss and pick a winner with a tie-break toward fewer epochs. Then add a second seed for the winner and see whether the gap survives.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

Why is the learning rate the first hyperparameter to tune?