EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

attribute
A named value attached to a span or trace, such as a model name or token count, that you can filter and group by.
cardinality
The number of distinct values a field can take. User ids make a field expensive to store and chart; a status code does not.
cost
The money a request spends, usually worked out from token counts and the price per token.
error taxonomy
A short, named list of failure classes, each with an owner and a rule for whether it can be retried.
feedback label
A defined judgement attached to a request after the fact, such as helpful, wrong or unsafe, with a rubric two people can apply alike.
latency
The time a request spends waiting and working, reported for each stage and for the whole.
p95
The cutoff that 95 out of 100 requests finish under, so a few very slow calls stay visible.
redaction
Removing or masking sensitive values before a record is stored, while keeping enough context to debug.
retention
How long stored records are kept before deletion, set by policy rather than by disk space.
sampling
Keeping only a chosen share of records, ideally with every failure kept and the totals corrected.
self time
The part of a span's duration spent in that step alone, after subtracting the time its children took.
span
One timed step inside a request, with a start, an end and the values it carried.
trace
One request's whole journey, with every step nested under the step that caused it.
trace id
A value shared by every record of one request, so pieces written by different services can be joined.

Observability for AI systems

Dashboards and alerts

A dashboard answers a question with an owner; an alert wakes someone who can act. Build both around symptoms, counts and a runbook, or the pager becomes noise.

Part 8 of 10

Intermediate16 min readBack to Courses

A dashboard answers a question

A dashboard is not a wall of charts; it is a small set of questions with an owner. Is the feature healthy, is it fast, is it affordable, and is it any good? Each panel should answer one of those at a glance, and a panel that answers none should be removed. The best dashboards are boring and short.

Start from the decision, not the metric. If the decision is whether to roll back a prompt, the panel needs the error rate by prompt version, the cost per success and the refusal rate on the same time window. A chart that cannot change a decision is decoration with a query cost.

Put the count and the window on every rate. A refusal rate of 40 percent on five requests and one on five thousand are different events, and a dashboard that hides the denominator invites the wrong page. The window belongs in the title, not in a tooltip nobody opens.

Alert on symptoms

An alert should fire on what the user experiences: errors, latency, cost per success, a refusal spike, a drop in task success. A CPU graph is a cause; nobody outside the team cares, and paging on it wakes someone for a problem that may never reach a user. Causes belong on the dashboard, symptoms in the pager.

A good alert has a threshold, a duration, a minimum volume and a runbook. The duration stops a single bad minute from paging; the volume stops a hundred percent error rate on two requests from paging; the runbook tells the responder what to check first. An alert without a runbook is a notification that says good luck.

Prefer error budgets to raw thresholds. A service that is allowed half a percent of failures can burn that budget slowly for a week or quickly in an hour, and the burn rate says which. Multi-window burn-rate alerts catch both the fast outage and the slow leak without a separate rule for each.

The two metrics every feature needs

from prometheus_client import Counter, Histogram

REQUESTS = Counter(
    "llm_requests_total", "Requests by feature and outcome", ["feature", "outcome"]
)
LATENCY = Histogram(
    "llm_latency_seconds", "End-to-end latency", ["feature"],
    buckets=(0.25, 0.5, 1, 2, 5, 10, 30),
)

REQUESTS.labels(feature="support", outcome="ok").inc()
LATENCY.labels(feature="support").observe(1.8)

Burn rate across three windows

Compare the three windows and see which one pages. Then add a five-minute window with forty requests and four failures and decide whether it should page.

Output
Click "Run" to execute your code

The columns an LLM feature needs

Four columns are specific to a model feature. Cost per successful answer turns a token bill into a unit the product can compare; refusal rate catches a prompt that became too cautious; truncation rate catches context that no longer fits; and cache hit rate explains a cost change that has nothing to do with traffic.

Quality needs a number even when it is approximate. A sampled judge score, a thumbs rate with its count, or the pass rate on a fixed regression set will do, as long as the definition and the sample size are on the panel. A dashboard with latency and cost but no quality shows a healthy system that answers badly.

Split by the dimensions that explain a change: feature, prompt version, model, tenant tier, retrieved-context size. A single aggregate hides the one tenant whose prompts are twice as long, and that tenant is usually the one who calls. The split is the difference between a number and a diagnosis.

An alert without a runbook is a page that says good luck. Link the dashboard panel and the first check in the alert body, or the responder starts from zero every time.

Alert fatigue is a failure mode

Every page has a cost: attention, sleep, and the willingness to respond to the next one. A team that gets twenty pages a week starts to triage by title, and the one that mattered is the one that looked like the others. Fewer, better alerts beat complete coverage.

Separate paging from ticketing. A page means a human must act now; a ticket means someone should look this week. Route cost drift, a slow quality decline and a single tenant’s bad week to a ticket, and keep the pager for user-visible failures with a runbook. The routing rule is part of the alert, not a habit.

Review alerts after every incident and every quiet month. An alert that never fires is either dead or has the wrong threshold; one that fires constantly is a missing automation or a bad threshold. Delete or fix it, and record why, so the next person does not add it back.

A rule table with a volume guard

Run the rules and see which features page and which only ticket. Then add a truncation rule with a minimum volume and a series that trips it.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What should an alert fire on?