Versioning and review

A prompt is source code

The prompt decides what the product does, so it changes for the same reasons code does: a bug report, a new policy, a model upgrade. When it lives as a string in a dashboard, that change has no diff, no reviewer and no way back. The next person to open it cannot tell whether a phrase was deliberate or left over from an experiment that ended months ago. The interface encourages the edit and records none of the reasoning behind it.

Versioning is the smallest fix. Every edit becomes a new numbered version with an author, a date and one sentence about why it exists. The old version stays readable, so an incident can be reproduced against the exact text that caused it rather than a reconstruction from memory. A rollback then becomes a single action instead of an emergency rewrite under pressure, which is when most second incidents happen. A version number is also the only handle a bug report can quote.

Immutable versions, movable labels

The useful split is between a version and a label. A version is immutable: once published, its text never changes, and its number only ever refers to that text. A label is a pointer — production, staging, canary — and moving a pointer is what a release means. The prompt a request used is then two facts, not one. Both live in the same store, and a version is never rewritten.

This is the same idea as a container tag that resolves to a digest. If editing version three in place were allowed, two requests a minute apart could run different prompts under the same name, and no log would show it. Immutability is what makes the version number in a log line trustworthy, and trust is the only reason to write it down. A request that cannot name its version is a request you cannot debug later.

Resolving a label with Langfuse

from langfuse import Langfuse

langfuse = Langfuse()

# The label resolves to an immutable version. Moving the label is the release.
prompt = langfuse.get_prompt("support-reply", label="production")
messages = prompt.compile(question=question, plan=plan)

reply = client.chat.completions.create(model="gpt-4o-mini", messages=messages)
print(prompt.version)  # log this beside the request id

What a review actually needs

Reviewing a prompt is not proofreading. The reviewer needs the diff, the reason, and the evidence: a fixed set of cases run against the old and the new version, with the outputs side by side. Without that table, approval is only an opinion about wording, and opinions do not catch a model that started refusing a category of question. The diff shows what the reader will see; the table shows what the model will do.

Keep the set small enough to read — twenty cases is usually enough to catch a regression in tone, format or refusal — and include the cases that failed before. A prompt change that fixes one case and silently breaks two is the normal failure, and only a comparison shows it. The reviewer should be able to say which cases changed and whether each change was intended. Attach the table to the change, not to a chat thread that will be lost.

Promote gradually, keep the receipt

A release should move a label, not edit text. Send a small share of traffic to the new version, compare the metrics that matter, then promote or roll back. The comparison is only honest if both versions run on the same inputs at the same time, so route by request id rather than by day, or the two samples will differ for reasons that have nothing to do with the prompt. A canary that shares no inputs with the control group measures the day, not the prompt.

Record every promotion in an append-only log: who moved which label, from which version to which, and why. That log answers the only question that matters during an incident — what changed, and when — without asking five people to check their memory. It also makes a canary reviewable after the fact, which is what turns a habit into a process. The log is short, boring, and the first thing you open when something breaks.