Supervised fine-tuning

Next-token prediction with a mask

The model predicts each token from the tokens before it, and the loss is the cross-entropy of the correct next token. For a chat example, the prompt tokens are context, not targets: you want the gradient to come from the assistant reply. Masking the prompt positions is the difference between teaching an answer and teaching the model to reproduce the question. The mask is a property of the data pipeline, not of the model architecture. Get it wrong and the loss improves while the behaviour gets worse.

The mask is applied to the labels, not to the input. The model still sees the whole conversation, because the reply depends on it; it simply is not penalised for failing to predict the user’s words. Get this wrong and the loss looks excellent while the model learns the wrong thing, which is one of the few failures that is invisible on a training chart. A model trained without a mask can answer a question by repeating it, which looks like fluency in a demo. The input and the label are the same tokens, shifted by one position.

One step is four operations

A step is a forward pass, a loss, a backward pass and an optimizer update. The batch size times the gradient accumulation steps is the effective batch; the learning rate schedule decides how far each update moves. Warmup prevents a large first step from destroying the pretrained weights, and decay lets the model settle instead of bouncing around the minimum. Warmup is cheap insurance against a first step that is much larger than the rest. The optimizer does not know about epochs; it only sees steps.

Checkpointing is part of the loop, not an afterthought. Save often enough to resume after a preemption, and evaluate the checkpoints rather than only the final one. The best checkpoint is often not the last, especially when the dataset is small and the model starts to memorise after the second epoch. A checkpoint that was never evaluated is a file, not a candidate. Evaluate the checkpoint you would actually deploy, not the one with the nicest curve.

A trainer in a few lines

from trl import SFTConfig, SFTTrainer

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    train_dataset=train,
    eval_dataset=val,
    args=SFTConfig(
        output_dir="out",
        num_train_epochs=3,
        learning_rate=2e-4,
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        max_length=1024,
    ),
)
trainer.train()

What the loss does not tell you

Training loss falls as the model fits the data, including the noise. A near-zero loss on a small set is a memorisation signal, not a quality signal. Validation loss is a better guide, and a task metric is better still: exact match, a rubric score, a pass rate, or whatever the product actually needs. The training curve is a diagnostic, not a scoreboard. A task metric can rise while the model becomes less useful in the cases the metric does not cover.

A narrow fine-tune can improve the target behaviour and hurt general instruction following at the same time. That trade is not visible in the training loss and often not visible in a task-only evaluation. It is why the next chapters add a general suite, a held-out set and a release gate rather than trusting one curve. A validation loss that turns up while training loss falls is the clearest signal you will get. A general suite is the only place that trade becomes a number.

Packing and truncation

Packing fills each training sequence with several short examples to reduce padding. It is efficient, and it is wrong without an end-of-sequence boundary: the model learns that the next example follows the previous answer. Insert the boundary token between examples and confirm the attention mask does not let one example see the next. Packing is an optimisation; correctness comes first. An end-of-sequence token is a boundary, not a separator to be skipped for speed.

Truncation has the same shape of risk. Cutting the assistant answer teaches incomplete replies; cutting the oldest turns preserves the target. Measure the truncation rate before training and treat a high rate as a dataset problem. A model trained mostly on fragments will produce fragments, and the loss will not complain. A fragment-trained model is especially good at sounding like it was interrupted. A high rate is a reason to fix the data, not to lower the loss threshold.