EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

adapter
A small set of trained numbers that changes how a fixed model behaves and can be added or removed without retraining it.
base model
The original model a tuning run starts from, left unchanged when only an adapter is trained.
catastrophic forgetting
When training on a narrow task makes a model worse at the general skills it already had.
checkpoint
A saved copy of a model's learned numbers at one point in training, kept so a run can be resumed or compared.
dataset
The collection of examples a model learns from, split into training, validation and test parts.
epoch
One full pass over the training examples; several passes let the model see the data more than once.
hyperparameter
A setting chosen before training, such as how fast the model learns, rather than one learned from the data.
leakage
When information from the test or future data reaches training, making the score look better than it is.
learning rate
How far the model's numbers move on each update; too high overshoots and too low barely moves.
LoRA
A method that leaves the model fixed and learns two small grids of numbers whose product is added to chosen parts.
loss
A number that measures how far the model's predictions are from the answers in the data, and that each update tries to lower.
preference tuning
Training a model to choose the better of two answers, rather than to copy one correct answer.
supervised fine-tuning
Training a model on example inputs and the answers you want it to produce for them.
validation set
Examples held back from training and used to choose settings and the stopping point.

Fine-tuning and LoRA

Preference tuning

SFT teaches an answer; a preference pair teaches a choice between answers. DPO turns that choice into a loss with a reference model and one leash called beta.

Part 8 of 10

Advanced17 min readBack to Courses

SFT imitates, preferences rank

Supervised fine-tuning teaches the model to produce the answer it was shown. Preference tuning teaches it to prefer one answer over another, which is a different signal: the pair says this one is better, not this is the only correct one. Many good answers exist, and a preference says how to choose among them.

The pipeline is usually SFT first, then preference tuning. SFT teaches the format and the task; preference tuning moves the distribution toward the answers people actually prefer, including tone, length and the willingness to say it does not know. Starting preference tuning on a base model that cannot follow the format spends pairs on a problem SFT solves cheaply.

RLHF does this with a reward model and a policy optimiser; DPO does it directly on pairs, with no separate reward model and a simpler loop. The trade is that DPO needs paired data rather than a scalar score, and its quality is bounded by the pairs. For most product teams the simpler loop is the right first attempt.

The shape of a pair

A preference example is a prompt, a chosen response and a rejected response. The two responses must answer the same prompt, or the comparison is meaningless; a pair that compares an answer to a different question teaches the model to prefer topics. Store the prompt id so the pair can be checked.

Collect pairs where the difference is real and nameable. A pair where both answers are fine is noise, and a pair where the annotator cannot say why is a coin flip with a label. Ask for a reason code: wrong fact, unsafe, too long, ignored the format, refused a legitimate request. The reason is what makes the dataset auditable.

Measure agreement before you train. Two annotators on the same prompt should pick the same winner most of the time; if they do not, the rubric is ambiguous and the pairs will teach the model to guess. Drop ties and contradictions rather than averaging them into a label that means nothing.

A DPO trainer on pairs

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM
from trl import DPOConfig, DPOTrainer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")
model = get_peft_model(base, LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"]))

trainer = DPOTrainer(
    model=model,
    ref_model=None,  # PEFT: the adapter-disabled base is the reference
    args=DPOConfig(beta=0.1, learning_rate=5e-6, num_train_epochs=1, output_dir="dpo-out"),
    train_dataset=pairs,
    processing_class=tokenizer,
)
trainer.train()
trainer.save_model("adapter-dpo")

Build pairs that agree

Run the agreement check and see which prompts are dropped. Then add a third annotator for p1 who disagrees and rerun the count.

Output
Click "Run" to execute your code

The DPO loss, in one line

DPO compares two log-ratios. For the chosen and rejected response, it takes the policy’s log-probability minus the reference model’s log-probability, and subtracts one from the other. The loss pushes the chosen margin up and the rejected margin down, with a beta that controls how far the policy may drift from the reference.

The reference model is the anchor. Without it, the loss can be reduced by making the rejected answer worse rather than the chosen one better, which is how a model learns to be strange. Beta is the leash: a small beta allows a large drift and a large beta keeps the policy close to the reference.

The loss is computed on the response tokens only, not the prompt, for the same reason SFT masks the prompt. A long prompt would otherwise dominate the log-probability and the pair would be decided by the question rather than the answer. The implicit reward is beta times the log-ratio, which is what you can plot.

Preference tuning finds shortcuts. If chosen answers are longer, the model learns longer; if a judge prefers lists, it learns lists. Audit the pairs for a confound before you blame beta.

Evaluate a preference tune

The primary metric is a win rate against the SFT model on held-out prompts, with the position of the two answers swapped to cancel position bias. A judge that always prefers the first answer will show a large effect that is entirely an artefact. Report the judge, the prompt and the number of comparisons.

Automatic judges are useful and biased. They favour length, formatting and their own model family, so pair the win rate with a human spot check on a stratified sample and with the task suite from the SFT stage. A preference win that loses a factuality point is not a win for the product.

Watch the slices: refusal rate on legitimate requests, length distribution, format compliance, and the general suite. Preference data is narrow, and the model can become better at being preferred while worse at being correct. The slices are how that shows up before a customer finds it.

The loss and the margin

Compare the loss for three pairs at three values of beta. Then add a pair where the policy already prefers the rejected answer and watch the loss.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What does a preference pair add over a supervised example?