What to record
A request that leaves no record cannot be debugged, priced or improved. The unit is one trace carrying the inputs, decisions, outputs and cost of a single request.
A request is a story, not a line
Most logging records that something happened. An AI request needs the story: what the user asked, which prompt version ran, what was retrieved, which tools were called, what came back, and how long each step took. Without that, a bad answer looks the same whether the cause was retrieval, a stale prompt or the model itself. The difference matters because each cause has a different fix and a different owner. The record is the difference between an anecdote and an incident report.
Write the record at the boundary where you can see the whole request. The model provider sees only its own call; your application is the only place that knows the question, the context and what the user did with the answer. That is why you instrument your code rather than relying on the dashboard the provider gives you. A provider dashboard is useful for its own latency and errors, and blind to everything before and after. Instrument the request handler, not the model client, so the rest of the tree has somewhere to hang.
Six fields that earn their keep
At minimum, record a request id, the prompt version, the model and its parameters, token counts for input and output, latency split by stage, and an outcome. The outcome is not only success or failure: it is whether the answer was accepted, corrected, retried or abandoned. Those six fields answer the first questions in any incident without a second query, and anything else is a bonus that should earn its storage. If a field cannot change a decision, it is telemetry for its own sake.
Add the ids of retrieved documents, not their full text. Ids let you reproduce the context later from a store that has its own retention rules, and they keep the trace small enough to search. Store the rank and score too, because they explain why a chunk was there. A trace carrying a megabyte of pasted context is a trace nobody can query, and a second copy of your corpus to protect. A document that has been re-indexed since the request is a different document, so the id should carry a version.
The fields, as span attributes
from opentelemetry import trace
tracer = trace.get_tracer("support-bot")
with tracer.start_as_current_span("chat") as span:
span.set_attribute("app.prompt_version", "support-v7")
span.set_attribute("gen_ai.request.model", "gpt-4.1-mini")
span.set_attribute("gen_ai.usage.input_tokens", 812)
span.set_attribute("gen_ai.usage.output_tokens", 96)
span.set_attribute("app.outcome", "accepted")One request, one record
Run the logger and read the record it writes. Then add a feedback field and a second request, and confirm both records share the same shape.
Record decisions, not just text
The output is the least interesting part of the record. The decisions around it explain it: which prompt version, which model, which retrieved chunks, which tool result, which guardrail fired. Two runs can produce the same sentence for different reasons, and the fix depends on which reason it was. A trace that stores only the sentence cannot tell them apart. It is a transcript, not an explanation. The same output under a different prompt version is a different experiment.
This is what makes a trace reviewable by someone who was not on call. A reviewer can see that the model followed the context correctly and the retriever returned an old policy page, which is a different ticket from a prompt that ignored the context. The record turns an argument about quality into a look at a cause. That is also how a new engineer learns what the system does, rather than what the diagram says.
Log the prompt version on every request, even while you are still editing prompts by hand. A trace without a version tells you the answer changed and not what changed.
Keep the record cheap and safe
Observability has a cost and a privacy surface. Cap field sizes, hash identifiers, and redact before the record leaves the process. Prompts and completions can contain personal data, so a trace store is a second copy of your users’ text and needs the same access rules as the primary store. A field you would not put in a support ticket does not belong in a trace. Redaction at the edge is cheaper than a cleanup after the export.
Retention should be deliberate: full prompts for days, aggregates for months. Most debugging happens in the first week, while cost and quality trends need the long window. Keeping everything forever is how a trace store becomes a liability, and how a routine query becomes an incident. Set the clock before the first busy day, not after. Deleting on a schedule is cheaper than defending a growing archive. A retention policy is a product decision, not a storage setting, and it should be reviewed like one.
A schema that rejects thin records
Run the validator over three records. Then add a record with an empty answer and confirm it is reported rather than stored.
Check what you learned
One question at a time. A wrong answer explains itself — take another go.
Part 1 of 5
What is the main reason to record the whole request rather than only the model call?