EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

attribute
A named value attached to a span or trace, such as a model name or token count, that you can filter and group by.
cardinality
The number of distinct values a field can take. User ids make a field expensive to store and chart; a status code does not.
cost
The money a request spends, usually worked out from token counts and the price per token.
error taxonomy
A short, named list of failure classes, each with an owner and a rule for whether it can be retried.
feedback label
A defined judgement attached to a request after the fact, such as helpful, wrong or unsafe, with a rubric two people can apply alike.
latency
The time a request spends waiting and working, reported for each stage and for the whole.
p95
The cutoff that 95 out of 100 requests finish under, so a few very slow calls stay visible.
redaction
Removing or masking sensitive values before a record is stored, while keeping enough context to debug.
retention
How long stored records are kept before deletion, set by policy rather than by disk space.
sampling
Keeping only a chosen share of records, ideally with every failure kept and the totals corrected.
self time
The part of a span's duration spent in that step alone, after subtracting the time its children took.
span
One timed step inside a request, with a start, an end and the values it carried.
trace
One request's whole journey, with every step nested under the step that caused it.
trace id
A value shared by every record of one request, so pieces written by different services can be joined.

Observability for AI systems

Privacy in traces

A trace is a second copy of the request, often longer-lived than the original. Classify every attribute, redact before export, and make deletion reach the trace store.

Part 9 of 10

Intermediate16 min readBack to Courses

A trace is a copy of the request

A trace is not metadata about a request; it is a second copy of the request with a schema. The prompt, the retrieved chunks, the tool arguments and the model output are all user text, and the trace store inherits every obligation the primary store has. Teams that would never log a card number will happily export one as a span attribute.

The copy is often longer-lived than the original. Application rows are deleted by a job; traces sit in a vendor with a retention setting nobody changed, and the vendor may be in another jurisdiction. The trace is frequently the largest and oldest collection of user text in the company.

Treat the trace as a data class, not as debugging output. That means an owner, a purpose, a retention period and an access rule, decided before the instrumentation ships. The first question is not what can we record but what must we never record.

Decide at the SDK, not in the backend

Redaction in the backend is redaction after the data has travelled, been stored and possibly been indexed. Do it in the SDK, before the span leaves the process: an allow-list of attribute keys, a redactor for free text, and a drop rule for anything unrecognised. The default for an unknown key should be drop, not keep.

An allow-list is stronger than a block-list because it fails closed. A new attribute added by a library is dropped until someone classifies it, which is the correct default. A block-list fails open, and the attribute nobody thought of is the one that carries the email.

Put the policy in one module and test it with the same cases as the primary store. If the trace redactor and the log redactor are two implementations, they will disagree, and the disagreement will be found by a customer rather than by a test. One function, two call sites.

A processor that drops what it was not told to keep

from opentelemetry.sdk.trace import SpanProcessor

SAFE_KEYS = {"gen_ai.request.model", "http.status_code", "feature", "tenant_tier"}


class AllowListProcessor(SpanProcessor):
    def on_start(self, span, parent_context=None):
        pass

    def on_end(self, span):
        for key in list(span.attributes):
            if key not in SAFE_KEYS:
                span.set_attribute(key, "[dropped]")

    def shutdown(self):
        pass

    def force_flush(self, timeout_millis=None):
        return True

Classify every attribute

Run the policy and confirm no raw value survives. Then add a session id attribute and decide whether hashing it is enough to export.

Output
Click "Run" to execute your code

Hashing, encryption and the join

A hash is a join key, not an anonymiser. An unkeyed SHA-256 of an email is reversible with a list of a billion addresses, and a customer list is smaller than that. Use a keyed hash with a secret held outside the trace store, or the hash is a slower spelling of the email.

Encryption is for values you need back. A support engineer debugging a specific ticket may need to see the prompt, and field-level encryption with an audited decrypt path gives them that without making the whole store readable. The key is the access control, so its use has to be logged.

Keep the join direction consistent. The trace should point at the subject with a pseudonym, and the mapping should live in one service with its own retention. If every store carries the email, every store has to be deleted separately and one of them will be missed.

Span names are attributes too. A span named after a route with an email in the path puts the email in the index, the dashboard legend and every screenshot. Name the route, not the value.

Retention, access and deletion

Use tiers rather than one retention setting. Metadata can live for months, redacted text for weeks, and full prompt capture for hours under a debug flag with consent. The tier is chosen per span, recorded as an attribute, and enforced by the exporter, not by a dashboard.

Access to traces is access to user text, so it needs the same controls as the primary store: named users, a reason, an audit entry per read, and no bulk export by default. A support tool that shows the full prompt to anyone with a login is a data leak with a search box. Every read should be attributable to a person and a ticket, and the read log should record the query rather than the text it returned.

Deletion has to reach the trace store and everything derived from it: the eval case built from a bad answer, the dashboard snapshot, the exported CSV. Aggregates can survive if they cannot be re-identified, which is a property to test rather than assume. The subject id is what makes the fan-out possible.

Delete a subject from traces and eval cases

Run the deletion and check that the eval case built from a deleted trace is gone too. Then add an export file that contains one of those traces and bring it into the fan-out.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What is a trace relative to the request it describes?