Observability for AI systems
See what the model did, and what it cost
- Intermediate
- 10 chapters
- 159 min read
Instrument an AI feature so a bad answer can be explained: traces and spans, prompt capture with redaction, latency by stage, cost per successful answer, sampling that keeps failures, an error taxonomy, feedback labels, dashboards and alerts, privacy in traces, and a capstone trace tree with a cost report.
What You'll Learn
- Record the fields that make a request explainable
- Build and read a trace tree with spans and self time
- Choose attributes that are safe for metrics and traces
- Capture prompts in layers, with redaction and retention
- Report latency by stage and cost per successful answer
- Sample traces without losing failures or breaking totals
- Classify errors by cause, owner and retryability
- Turn feedback into defined labels and regression cases
- Build dashboards and alerts that page on symptoms, not causes
- Redact, retain and delete traces without losing the join
Prerequisites
- Comfortable with Python dictionaries and functions
- Some experience running a web service is useful
- No tracing vendor or API key is needed — the examples use plain records
Vocabulary
The terms the chapters assume, in plain words.
- attribute
- A named value attached to a span or trace, such as a model name or token count, that you can filter and group by.
- cardinality
- The number of distinct values a field can take. User ids make a field expensive to store and chart; a status code does not.
- cost
- The money a request spends, usually worked out from token counts and the price per token.
- error taxonomy
- A short, named list of failure classes, each with an owner and a rule for whether it can be retried.
- feedback label
- A defined judgement attached to a request after the fact, such as helpful, wrong or unsafe, with a rubric two people can apply alike.
- latency
- The time a request spends waiting and working, reported for each stage and for the whole.
- p95
- The cutoff that 95 out of 100 requests finish under, so a few very slow calls stay visible.
- redaction
- Removing or masking sensitive values before a record is stored, while keeping enough context to debug.
- retention
- How long stored records are kept before deletion, set by policy rather than by disk space.
- sampling
- Keeping only a chosen share of records, ideally with every failure kept and the totals corrected.
- self time
- The part of a span's duration spent in that step alone, after subtracting the time its children took.
- span
- One timed step inside a request, with a start, an end and the values it carried.
- trace
- One request's whole journey, with every step nested under the step that caused it.
- trace id
- A value shared by every record of one request, so pieces written by different services can be joined.
Course Content
- 01What to recordNot started
A request that leaves no record cannot be debugged, priced or improved. The unit is one trace carrying the inputs, decisions, outputs and cost of a single request.
- 02Spans and tracesNot started
A trace is a tree of timed operations, not a list of timings. The parent links are what turn a pile of durations into a diagnosis you can act on.
- 03Attributes and prompt captureNot started
Attributes are the columns you query and the labels you must keep bounded. Prompt capture is a separate decision with its own privacy and retention rules.
- 04Latency and costNot started
Latency is a sum of stages and cost is a product of tokens and prices. Report the stage and the success, or the number will point at the wrong fix.
- 05SamplingNot started
Keeping every trace is a development habit, not a production budget. Sampling is a policy: keep the failures, sample the successes, and weight the totals.
- 06An error taxonomyNot started
A provider timeout, a schema violation and a refusal are different failures with different owners. Classify them, and decide what is retryable before the incident.
- 07Feedback loops and labelsNot started
A thumbs-down is an event, not a metric. Turn feedback into defined labels and regression cases, or the dashboard grows while the product does not.
- 08Dashboards and alertsNot started
A dashboard answers a question with an owner; an alert wakes someone who can act. Build both around symptoms, counts and a runbook, or the pager becomes noise.
- 09Privacy in tracesNot started
A trace is a second copy of the request, often longer-lived than the original. Classify every attribute, redact before export, and make deletion reach the trace store.
- 10Capstone: traces and a cost reportNot started
One file that turns span records into a readable tree, a cost per feature and a list of expensive failures. Every idea in the course, once.