Cost and caching

The arithmetic is simple

Price is quoted per million tokens, separately for input and output, and output usually costs several times more. A request with a long prompt and a short answer is dominated by input; a request that generates a page of text is dominated by output. The ratio is the first thing to compute, because it tells you which half of the product to change.

That ratio tells you where to optimise. Shortening a system prompt saves input tokens on every call; asking for a shorter answer saves output tokens, which are the expensive ones. Both are product changes as much as engineering ones. A prompt that says “be concise” is a cost decision disguised as a style instruction.

The arithmetic also explains why a small model with a long prompt can be cheaper than a large model with a short one, and why the opposite is often true for output-heavy work. Do the multiplication for your own traffic instead of reasoning from the sticker price. The sticker price is per token; the product is per request, per user and per month. A model that is twice as expensive per token but answers in half the tokens is the cheaper one.

Caching is a prefix match

Prompt caching stores the model’s internal state for a prefix of the request and reuses it when the same prefix appears again. It only works from the start: a stable system prompt and a stable set of documents can be cached, and a question placed before them breaks the match. The cache is not a database; it is a shortcut for a prefix the provider has seen before.

The discount is large and the requirement is strict. Put everything stable first, everything variable last, and measure the hit rate rather than assuming it. A cache that never hits is a more expensive request with extra steps. The hit rate belongs on the same dashboard as cost, because the two move together.

Cost from the response

from litellm import completion, completion_cost

response = completion(
    model="gpt-4.1-mini",
    messages=messages,
)

print(completion_cost(completion_response=response))
print(response.usage)

Route by difficulty

Not every request needs the largest model. A classifier, an extractor or a short rewrite often runs on a small model at a fraction of the price, and the router that decides can be a rule rather than a model. Start with the rule: length, language, task type and whether the request needs a tool are all cheap signals.

Routing has a quality cost that must be measured, not assumed. Send a sample of routed traffic to the large model and compare; if the small model’s answers are indistinguishable on that sample, the saving is real. If they are not, the saving is a refund request waiting to happen. The sample should be stratified by task, not drawn from the easiest requests.

The bill is a product metric

Cost per active user, cost per conversation and cost per successful task are product numbers. They move when the prompt changes, when retrieval returns more documents, and when a retry policy gets looser — often without anyone noticing. A dashboard that only shows total spend hides which of those three moved.

Set a budget per request and alert when the p95 crosses it. The alert is what turns an optimisation project into a guardrail, and it catches the regression that a code review would not. The budget should be a number the product can defend, not the current average rounded up.

The guardrail should also have an owner. A budget that nobody watches is a number in a config file; a budget with an alert, a dashboard and a named person is a control. When the p95 crosses the line, the question is not “which model is expensive?” but “which feature changed?”, and the tags on the metric are what answer it.