EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

attribute
A named value attached to a span or trace, such as a model name or token count, that you can filter and group by.
cardinality
The number of distinct values a field can take. User ids make a field expensive to store and chart; a status code does not.
cost
The money a request spends, usually worked out from token counts and the price per token.
error taxonomy
A short, named list of failure classes, each with an owner and a rule for whether it can be retried.
feedback label
A defined judgement attached to a request after the fact, such as helpful, wrong or unsafe, with a rubric two people can apply alike.
latency
The time a request spends waiting and working, reported for each stage and for the whole.
p95
The cutoff that 95 out of 100 requests finish under, so a few very slow calls stay visible.
redaction
Removing or masking sensitive values before a record is stored, while keeping enough context to debug.
retention
How long stored records are kept before deletion, set by policy rather than by disk space.
sampling
Keeping only a chosen share of records, ideally with every failure kept and the totals corrected.
self time
The part of a span's duration spent in that step alone, after subtracting the time its children took.
span
One timed step inside a request, with a start, an end and the values it carried.
trace
One request's whole journey, with every step nested under the step that caused it.
trace id
A value shared by every record of one request, so pieces written by different services can be joined.

Observability for AI systems

Capstone: traces and a cost report

One file that turns span records into a readable tree, a cost per feature and a list of expensive failures. Every idea in the course, once.

Part 10 of 10

Advanced18 min readBack to Courses

One file, two questions

The capstone answers two questions from the same records: where did the time go, and what did it cost. The first is a tree built from parent ids; the second is a sum of tokens times prices, grouped by the attributes that explain the spend. Neither needs a vendor SDK, which is the point: the data model is the skill. Plain dictionaries keep every field visible, including the ones a library would rename. The code is short enough to read in one sitting, and every number is visible.

The report is what a team reads on Monday. It should fit on a screen, name the traces that are worth opening, and show the count behind every rate. A report that requires a dashboard to interpret is a report that will be looked at once. A reader should not need a second window open to understand the first. The sections are ordered by the question they answer, not by the order the data arrived.

The shape of the report

Per trace, print the tree with duration and self time, then the total cost. Per feature and prompt version, print requests, p95 latency, cost, cost per success and error rate. Finish with the most expensive failures and the feedback attached to them, because a costly failure is the one worth fixing first. The tree is for the single request, the table is for the comparison, and the failure list is for the next hour of work. A reader should be able to stop after any section and still have a complete answer.

Keep the sections in that order. The order is a reading order: case, population, then the queue for the next hour. Mixing them produces a page that is interesting and not actionable. Put the count next to the rate and the window in the header; those two choices prevent most misreadings. A report that changes its price table between rows is not a report; it is a set of unrelated estimates.

Export the spans you just built

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
# In production, replace ConsoleSpanExporter with your trace backend's exporter.

A tree with a bill attached

Print both trace trees and the cost per span, then add a third trace with a retry under generate. Confirm the retry appears in the tree and in the total.

Output
Click "Run" to execute your code

Make the numbers comparable

A report is only useful if two rows mean the same thing. Use one price table, one time window and one definition of success across every feature. A 24-hour row compared with a seven-day baseline is not a comparison; it is two different questions sharing a column. A row without its window is a number that invites the wrong comparison.

Record the window and the price version in the report header. When someone asks why the number differs from last week, the header answers before the investigation starts. Reproducibility is a feature of the report, not a property of the dashboard. The header is also the first thing to check when a number looks impossible. A number that cannot be reproduced is a rumour with a decimal point.

Print the counts next to every rate. A 100 percent error rate on two requests is not a crisis; a 2 percent rate on 50,000 requests is, and the percentage alone cannot tell them apart.

What the capstone leaves out

A production system adds exporters, storage, retention, dashboards and alerts. Each is an adapter around the same records, not a new model. The exporter is the clearest example: the same spans can go to a console, a file or a hosted backend without changing the instrumentation. In production, it is the only line that changes. The hard part is deciding what to record and what each field means; the plumbing can be replaced without changing the questions. Retention is the one adapter with a legal dimension: a trace store is a copy of user text and inherits its obligations. The rest is a deployment choice.

The one thing not to leave out is the join key. Every cost row, feedback event and eval case should point at a trace id. Without it, the tree, the bill and the complaint are three unrelated datasets that happen to describe the same request. The id is small, stable and already in every record; there is no excuse for a report without it. The report is only as strong as that link.

The Monday report

Group the traces by feature and print requests, cost, cost per success and error rate. Then add a negative feedback event to the most expensive failure and show it in the list.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What two questions does the capstone answer from the same records?