Latency and cost

Latency is a sum, not a number

A request spends time in a queue, in retrieval, waiting for the first token, generating the rest, and in post-processing. Reporting one total hides which part changed. A p50 of two seconds can sit next to a p95 of fourteen, and the users who leave are usually in the tail. The fix for a slow retriever is different from the fix for a slow model, and a single average cannot tell you which one you have. A queue delay is often the largest part on a busy day and the easiest to mistake for model latency.

Streaming changes the number that matters. Time to first token decides whether the interface feels alive; total generation time decides when the answer is complete. Measure both, and keep them separate in the trace. A streaming interface that shows the first sentence in 400 milliseconds feels faster than a batch call that returns in 1.2 seconds, even when the batch call finishes sooner. An average over a mixed workload of short and long answers describes neither.

Cost is a product

Cost is input tokens times the input price plus output tokens times the output price, with cached input priced separately and tool or retrieval compute added. Prices are per million tokens and they change, so keep the table in code with a date and review it like a dependency. A cached input token is often half the price or less; a batch API can be cheaper again. The table should encode those differences, not a single blended rate. A pricing change and a usage change look identical in a total, and the table is how you tell them apart.

Cost per request is not unit economics. Cost per successful answer is: divide by the requests that produced an accepted result, not by all requests. A feature that fails half the time can look cheap per call and expensive per success, which is the number the budget owner cares about. Include the cost of retries and failed calls in the numerator, or the number flatters the feature.

A histogram and a counter

from opentelemetry import metrics

meter = metrics.get_meter("support-bot")
latency = meter.create_histogram("gen_ai.client.operation.duration", unit="s")
cost = meter.create_counter("app.cost.usd")

latency.record(1.84, {"gen_ai.request.model": "gpt-4.1-mini", "app.stage": "generate"})
cost.add(0.0004, {"app.feature": "support", "app.prompt_version": "support-v7"})

Attribute cost where decisions are made

A total bill is not actionable. A bill by feature, tenant, prompt version and model tells you that a verbose prompt costs more than the model upgrade, or that one customer is responsible for a third of the spend. Attribute at the point where the decision was made, not at the billing export. A prompt version that added three examples may have doubled input tokens; the report should show that before the next prompt review.

Cache hits change the effective price, so record them rather than inferring them later. A cached prompt can be an order of magnitude cheaper, and a report that ignores the discount will recommend the wrong optimisation. The same applies to retries: a retried call is paid for twice. Retries are also a latency problem, not only a cost problem, so the same attribute should appear on the span and the bill. A cache hit rate without the token count is a percentage that cannot be turned into money.

Percentiles with small n

A percentile is only as good as the sample behind it. With twenty requests, p95 is effectively the maximum, and a single slow outlier moves it by seconds. Report the count next to every percentile, aggregate over a window, and compare distributions rather than single numbers. A p95 over a day of traffic is a different statistic from a p95 over the last five minutes, and both are useful for different questions.

Nearest-rank is the simplest definition and the one to use when you compute it yourself: sort, take the value at the ceiling of p percent of n. Different libraries interpolate differently, so a dashboard and a script can disagree on the same data. Pick one definition and write it down. When someone asks why the dashboard disagrees with the invoice, the answer is usually a different window, a different percentile or a different definition of success.