EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

adapter
A small set of trained numbers that changes how a fixed model behaves and can be added or removed without retraining it.
base model
The original model a tuning run starts from, left unchanged when only an adapter is trained.
catastrophic forgetting
When training on a narrow task makes a model worse at the general skills it already had.
checkpoint
A saved copy of a model's learned numbers at one point in training, kept so a run can be resumed or compared.
dataset
The collection of examples a model learns from, split into training, validation and test parts.
epoch
One full pass over the training examples; several passes let the model see the data more than once.
hyperparameter
A setting chosen before training, such as how fast the model learns, rather than one learned from the data.
leakage
When information from the test or future data reaches training, making the score look better than it is.
learning rate
How far the model's numbers move on each update; too high overshoots and too low barely moves.
LoRA
A method that leaves the model fixed and learns two small grids of numbers whose product is added to chosen parts.
loss
A number that measures how far the model's predictions are from the answers in the data, and that each update tries to lower.
preference tuning
Training a model to choose the better of two answers, rather than to copy one correct answer.
supervised fine-tuning
Training a model on example inputs and the answers you want it to produce for them.
validation set
Examples held back from training and used to choose settings and the stopping point.

Fine-tuning and LoRA

Serving adapters

An adapter is a deployment artefact: base digest, tokenizer, template and config. Merge it or swap it, route it from identity, and version every cache key.

Part 9 of 10

Advanced17 min readBack to Courses

The adapter is a deployment artefact

An adapter file is meaningless without the base model, the tokenizer and the chat template it was trained against. Serving it is therefore a four-part deployment, and the registry entry should name all four plus the training config and the evaluation that approved it. An adapter id alone is a filename, not a version.

Pin the base by digest, not by a tag that can move. A provider that updates the model behind the same name will change the behaviour of your adapter without a deploy, and the trace will still say the adapter version you tested. The base is part of the artefact even though you did not train it.

Keep the registry append-only. A version that was served stays in the record even after it is retired, because a trace from last month points at it and an incident review needs to load it. Deleting a version is deleting evidence.

Merge or swap

Merging adds the low-rank update into the base weights and produces an ordinary model file. It removes the adapter’s extra matrix multiplication and the per-request lookup, at the cost of one full model copy per variant. That is the right trade when a single behaviour serves all traffic.

Swapping keeps one base in memory and loads adapters beside it. Serving frameworks batch requests from different adapters against the same base, which is how one GPU serves many tenants. The cost moves to adapter memory, load time and a cache of recently used adapters.

The memory arithmetic decides the design. Each adapter is roughly rank times the sum of the adapted dimensions times two bytes, per module, and the KV cache needs its own headroom. A GPU that fits the base and four adapters may not fit the base, eight adapters and a long-context batch.

One base, many adapters

from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest

llm = LLM(model="Qwen/Qwen2.5-1.5B-Instruct", enable_lora=True, max_loras=4)
request = LoRARequest("support-v3", 1, "adapters/support-v3")
llm.generate(["Where is my refund?"], SamplingParams(max_tokens=128), lora_request=request)

What actually limits adapter count

Print the size of each adapter and the binding limit. Then add a rank-128 adapter and see whether memory or the framework slot count binds first.

Output
Click "Run" to execute your code

Routing and isolation

A multi-tenant service chooses an adapter from the authenticated tenant, never from a request field. The tenant comes from the session or the API key, the registry maps it to a version, and the request is rejected if the mapping is missing. A client-supplied adapter name is a request to run someone else’s weights, and the registry is the authorisation table for weights.

Isolation is a correctness property, not a performance one. If adapter state can leak between batched requests, one tenant’s tone appears in another tenant’s answer. Test it with a canary prompt whose output differs by adapter, and assert the cross-tenant pair never matches. The test belongs in the serving layer, not in the fine-tuning notebook.

Always have a fallback. If an adapter fails to load, the service can answer from the base model with a recorded degradation, or refuse; either is better than a 500. The fallback is a product decision, so name it in the registry entry rather than discovering it during a deploy.

A mismatched chat template does not throw. It produces a plausible answer with the wrong special tokens, and the evaluation you ran offline no longer describes what is served. Version the template with the adapter and compare one rendered prompt at startup.

Rollout, rollback and cache

Roll out an adapter like any model change: a canary slice, a fixed window, the task suite and the general suite, and a gate that can say no. The canary should be large enough to see a rare failure and small enough to contain it. Ten percent for an hour is a start, not a law.

Put the adapter version in every trace, next to the base model digest and the template version. When the feedback turns, the join tells you which of the three changed. Without it, the review compares a prompt change with a weights change and learns nothing.

Invalidate the caches on deploy. A response cache keyed by prompt but not by adapter version will serve the old behaviour after the rollback, which looks like the rollback failed. The same applies to embeddings and to any evaluation cache used by the gate. The cache key is part of the version.

Route by identity, not by request

Run the requests and check that each tenant reaches its own adapter or a named fallback. Then raise the acme canary to half and confirm the split follows the roll value.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What must be stored with an adapter to serve it reproducibly?