Prompts across models

Same words, different program

A prompt is not executed by a specification. It is read by a model whose instruction tuning, chat template and tokenizer were chosen by someone else. The same sentences can produce a careful refusal on one model and a confident invention on another, and neither is a bug in the prompt as written. The difference is not a defect in either one; it is a difference in what each was trained to prefer.

That is why a prompt tuned on one model is a draft rather than an asset. It encodes assumptions about how that model was trained: how much it likes numbered steps, whether it honours a system role, how it formats JSON. Moving it is not copying a string, it is porting a program to a runtime with different defaults. The port is finished when the new model passes the cases the old one passed, not when the text compiles. Until then it is a hypothesis about the new runtime.

What actually differs

Four differences explain most surprises. The tokenizer decides what a prompt costs and how much of the context window it fills, and the same paragraph can be twenty percent longer on one model. The chat template decides whether a system message is a real channel or is folded into the first user turn. A prompt that fits one window can overflow another before the conversation even starts.

The output contract differs too: one model has a JSON mode, another only follows a schema in the prompt, and a third wraps JSON in prose unless told twice. Tool calling is a fourth: the schema shape and the way arguments arrive vary, so a tool description that works on one provider can be silently ignored by another. Each of those is a place where the same instructions can be heard differently, and a tool call that arrives as a string rather than an object is the kind of break that only shows up in production.

One call shape, several providers

from litellm import completion

# The model string selects the provider; the request stays the same.
reply = completion(
    model="anthropic/claude-sonnet-4-5",
    messages=[
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": question},
    ],
    max_tokens=400,
    temperature=0,
)
print(reply.choices[0].message.content)

Write for the weakest link

The portable prompt states its contract twice: once as an instruction and once as a short example of the output. It avoids tricks that only one model was trained to reward, such as a particular phrase for reasoning, and it says what to do rather than only what to avoid. A model that has never seen your favourite incantation still understands “return one JSON object with these keys”. Concrete nouns and a named format travel further than adjectives about quality.

Put durable rules in the system message and the task in the user message, because that split survives the most templates. Where a model has no system channel, fold the rules into the first user turn with a clear heading, and keep the same wording so the two versions can be compared rather than merely resemble each other. A comparison between two renderings is only meaningful when the rules are identical.

Port by measuring, not rewriting

Start with the current prompt unchanged and record where the new model fails: a missing key, a longer answer, a refusal, a tool it never calls. Each failure points at one part of the prompt, and the fix should touch that part only. A full rewrite throws away everything the old prompt learned and gives you no way to tell which edit mattered. Small edits are also reviewable: a reviewer can see exactly which sentence changed.

Keep a per-model variant only when the difference is real and stable, and version it separately so the two do not drift by accident. Most of the time the honest answer is one prompt with a capability check at the edge, and a note in the registry saying which models it was verified against and when. Write the verified models and the date in the registry entry, so the claim ages visibly instead of becoming folklore.