EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

attribute
A named value attached to a span or trace, such as a model name or token count, that you can filter and group by.
cardinality
The number of distinct values a field can take. User ids make a field expensive to store and chart; a status code does not.
cost
The money a request spends, usually worked out from token counts and the price per token.
error taxonomy
A short, named list of failure classes, each with an owner and a rule for whether it can be retried.
feedback label
A defined judgement attached to a request after the fact, such as helpful, wrong or unsafe, with a rubric two people can apply alike.
latency
The time a request spends waiting and working, reported for each stage and for the whole.
p95
The cutoff that 95 out of 100 requests finish under, so a few very slow calls stay visible.
redaction
Removing or masking sensitive values before a record is stored, while keeping enough context to debug.
retention
How long stored records are kept before deletion, set by policy rather than by disk space.
sampling
Keeping only a chosen share of records, ideally with every failure kept and the totals corrected.
self time
The part of a span's duration spent in that step alone, after subtracting the time its children took.
span
One timed step inside a request, with a start, an end and the values it carried.
trace
One request's whole journey, with every step nested under the step that caused it.
trace id
A value shared by every record of one request, so pieces written by different services can be joined.

Observability for AI systems

Sampling

Keeping every trace is a development habit, not a production budget. Sampling is a policy: keep the failures, sample the successes, and weight the totals.

Part 5 of 10

Intermediate15 min readBack to Courses

You cannot keep every trace

Trace volume grows with traffic, and every span carries text, attributes and a retention cost. Sampling is a policy for choosing which requests to keep, and it has to be designed rather than switched on. The default of keeping everything works in development and becomes a budget line in production, usually without anyone deciding it should. A trace with a long prompt can be larger than the request it describes, which is why content capture has its own budget.

Head sampling decides when the request starts: cheap, no buffering, but it cannot know whether the request will fail. Tail sampling waits for the trace to finish: it can keep every error and every slow request, at the cost of holding traces in memory. Most systems need both a cheap default and a tail rule for the cases that matter. The two decisions are made in different places and should be documented together. A tail policy needs the whole trace in one place; a collector that sees spans one at a time can only approximate it.

Keep the failures

A sample that drops failures makes the error rate look better than it is. Always keep errors, policy blocks, slow traces and traces with negative feedback; sample the successes. The kept set is then a biased sample by design, which is fine for inspection and wrong for statistics. A dashboard that treats it as representative will report a healthier system than the one users see. An error dashboard built on that set answers what broke, not how often it breaks.

Deterministic sampling by trace id keeps all spans of a request together and makes the decision reproducible. Hashing the id and taking a modulo is enough; a random number at each span would split a trace and produce orphans. The same hash also lets you reproduce a customer report: given the trace id, you know whether it should have been kept before you search for it. The hash should be stable across services and versions; a changing hash silently changes the sample.

A ratio sampler that respects the parent

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased

provider = TracerProvider(sampler=ParentBased(TraceIdRatioBased(0.1)))
# ParentBased keeps a child when its parent was sampled, so a trace
# is not split across services that each roll their own dice.

Keep every failure, sample the rest

Run the sampler over ten traces. Confirm every error and slow trace survived, then compute the error rate in the kept set and compare it with the true rate.

Output
Click "Run" to execute your code

Sampling breaks totals unless you weight

If you keep one in five successes, each kept success represents five. Multiply by the inverse probability to estimate totals, and report the estimate with its uncertainty. The estimate is fine for a trend and too noisy for an invoice. A weighted number without the weight is a number that cannot be checked by anyone else. Report the weight next to the estimate so a reader can reconstruct the arithmetic without the code.

For billing and SLOs, use unsampled counters rather than traces. A counter incremented on every request costs almost nothing and does not need the text. Traces answer why; counters answer how many. Mixing the two is how a team ends up arguing about a 40 percent error rate that was really a sampling artefact. A counter can be incremented before the response is sent, which makes it cheaper than a span and harder to lose.

Never compute the latency distribution from a sample selected for being slow. Tail sampling is for inspection, not for SLOs, and a p95 built from slow traces is a p95 of slow traces.

Sample prompts separately

Trace sampling and content capture are different decisions. You can keep every span and one percent of prompts, or keep full prompts for every failure and none for successes. The second is often the right default: the expensive, sensitive part is the text, while the shape of the request is cheap. Treating them as one switch forces a choice between cost and debuggability. A prompt-capture policy can be turned off per tenant without changing the trace tree or the cost report.

Record the sampling rate as an attribute on every trace. A report that mixes sampled and unsampled rows without the rate will produce a number that cannot be reproduced. The rate is part of the data, not a dashboard setting. When the policy changes, the old traces should still say which policy kept them. A trace kept under an old rate is still valid data; a trace labelled with the wrong rate is not.

Weight the sample back to a total

Estimate total cost and error rate from a sample that keeps all errors and one in five successes. Compare the estimate with the true values and add a second feature with a different rate.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What is the difference between head and tail sampling?