EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

attribute
A named value attached to a span or trace, such as a model name or token count, that you can filter and group by.
cardinality
The number of distinct values a field can take. User ids make a field expensive to store and chart; a status code does not.
cost
The money a request spends, usually worked out from token counts and the price per token.
error taxonomy
A short, named list of failure classes, each with an owner and a rule for whether it can be retried.
feedback label
A defined judgement attached to a request after the fact, such as helpful, wrong or unsafe, with a rubric two people can apply alike.
latency
The time a request spends waiting and working, reported for each stage and for the whole.
p95
The cutoff that 95 out of 100 requests finish under, so a few very slow calls stay visible.
redaction
Removing or masking sensitive values before a record is stored, while keeping enough context to debug.
retention
How long stored records are kept before deletion, set by policy rather than by disk space.
sampling
Keeping only a chosen share of records, ideally with every failure kept and the totals corrected.
self time
The part of a span's duration spent in that step alone, after subtracting the time its children took.
span
One timed step inside a request, with a start, an end and the values it carried.
trace
One request's whole journey, with every step nested under the step that caused it.
trace id
A value shared by every record of one request, so pieces written by different services can be joined.

Observability for AI systems

Spans and traces

A trace is a tree of timed operations, not a list of timings. The parent links are what turn a pile of durations into a diagnosis you can act on.

Part 2 of 10

Intermediate16 min readBack to Courses

A trace is a tree

A trace is one request, and a span is one timed operation inside it. The root span covers the request; its children cover retrieval, the model call, each tool call and post-processing. Each span records a start, an end and a parent, so the set forms a tree rather than a flat list. A span without a parent is a root, which is correct only for the request itself. The tree is the point: it shows what caused what, not only what was slow. A span that starts before its parent is a clock or propagation bug, not a scheduling miracle.

A flat log of timings can tell you that a request took 1.8 seconds. A tree tells you that 1.4 of those seconds were one model call, that retrieval took 280 milliseconds, and that the tool call at the end was waiting on the same connection pool as the retriever. The shape is the diagnosis. Two traces with the same total can have different shapes, and only one of them has an obvious fix.

The parent link is the hard part

Spans start in different places: a web handler, a thread pool, an async task, a queue consumer. If a worker starts a span without the current context, that span becomes a second root. The trace then looks like several unrelated requests, and the time spent waiting between them disappears. Context propagation is the unglamorous part that makes the tree true. Pass the trace id and the parent span id explicitly at every boundary you own.

Most tracing libraries propagate context automatically inside one process, and stop at a thread or a queue unless you carry it. Put the ids in the job payload, restore them in the worker, and start the worker span as a child. The same failure appears when a background job is enqueued during a request and later writes its span to a new trace. A test that asserts one root per request catches the day this breaks. The id in the payload also lets a retry join the original trace instead of starting a second one.

Nesting spans around the work

from opentelemetry import trace

tracer = trace.get_tracer("support-bot")

with tracer.start_as_current_span("answer") as root:
    with tracer.start_as_current_span("retrieve") as span:
        docs = retriever.search(question)
        span.set_attribute("retrieval.doc_count", len(docs))

    with tracer.start_as_current_span("generate") as span:
        reply = model.complete(question, docs)
        span.set_attribute("gen_ai.usage.output_tokens", reply.usage.output)

Build the tree and find the path

Print the trace as a tree with self time, then follow the critical path. Add a retry span under the model call and see how the path changes.

Output
Click "Run" to execute your code

Name for grouping, attribute for finding

A span name is an index key, so keep it low-cardinality: retrieve, generate, tool, validate. Put the variable part in attributes: document ids, tool name, model, tenant. A span named after a user id creates one bucket per user and makes the latency chart useless. High cardinality belongs in traces, where you query individual rows, not in metrics, where every value multiplies the series count.

Attributes are also how you join a trace to the rest of your system. A deployment id, a prompt version and a feature flag turn an isolated request into a row you can compare across releases. Decide the names once and keep them stable; a renamed attribute is a silent break in every dashboard. That is what lets you ask whether p95 rose after the deploy, without opening a single log line. Without a deploy marker, a latency change has no timestamp to compare against.

A span that wraps the whole request and nothing else measures the wall clock and hides the cause. The useful spans are the children, one per operation you could optimise.

Self time and the critical path

A span’s duration includes its children, so the root always looks slow. Subtract the children to get self time, which is where the span itself spent time: a network wait, a tokenizer, a database query. A span with a long duration and near-zero self time is a container, not a cause. Children can overlap if you are not careful; then self time is a lower bound, and the overlap is a bug in the instrumentation.

The critical path is the chain of spans that determines the total. Optimising a span that is not on it saves nothing, which is why a fast retriever can leave a slow request unchanged. Walk from the root and follow the child with the largest duration until you reach a leaf; that chain is the one to fix. Record the path in the trace summary so a reviewer does not have to reconstruct it by hand.

Repair a broken trace

A worker lost its parent context, so one span arrived as a second root. Count the orphans, attach them to the request root, and print the tree again.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What is the difference between a trace and a span?