Callbacks and tracing

The chain is a black box

A chain is a few calls in a row, and when the answer is wrong the useful question is which call produced the bad input. Print statements answer it for one run and then become noise. Callbacks are the same idea with a seam: the framework announces that a step started, ended or failed, and your object decides what to do with the announcement. The alternative is a debugger that only works on your machine, for the one request you can reproduce.

The announcement carries the run id, the parent run id, the inputs and the output. Those four fields are enough to rebuild a tree, which is what a trace is. Without the parent id you have a list of events; with it you have the shape of the execution, and the shape is what tells you where the time went and which step was retried. A trace without timings tells you the order; with timings it tells you the cost.

A handler is an object with hooks

In LangChain a handler subclasses BaseCallbackHandler and overrides methods such as on_llm_start, on_llm_end, on_tool_start, on_tool_error and on_chain_end. The base class supplies empty implementations, so a handler implements only the hooks it cares about. The same object can count tokens, time calls and report failures without touching the chain itself. Because the base class is permissive, a handler can start as three lines and grow only when a question needs an answer.

Two rules keep handlers harmless. They must be cheap, because they run inline with the request, and they must not raise, because an exception in a handler can fail the call it was observing. Send the event to a queue and let another process write it, and wrap the body in a try/except that records the handler’s own failure and moves on. A handler that writes to a database on every token is the classic way to make a fast chain slow.

A handler that counts tokens

from langchain_core.callbacks import BaseCallbackHandler


class TokenCounter(BaseCallbackHandler):
    def on_llm_end(self, response, **kwargs):
        usage = (response.llm_output or {}).get("token_usage", {})
        print("tokens:", usage.get("total_tokens", "unknown"))

    def on_tool_error(self, error, **kwargs):
        print("tool failed:", type(error).__name__)


chain.invoke({"topic": "retries"}, config={"callbacks": [TokenCounter()]})

What to record

A useful trace records more than text. For each run: the name, the parent, the start and end time, the model and prompt version, the token usage, the status and any error. Inputs and outputs are valuable for debugging and dangerous for privacy, so redact them before they leave the process, not after they are stored. The run id is what ties a log line, a metric and a user report to the same execution.

Traces, metrics and logs answer different questions. A trace explains one request; a metric says how often something happens; a log records a discrete event for search. Sending everything to one store makes all three worse, and it makes the bill larger without making the answer clearer. Pick the question first, then the store. A trace that is never queried is a cost with no reader, so decide who will look and what they will ask.

Tracing in production

In production, sample. Tracing every request is affordable for a small service and expensive for a large one, and the thousandth identical success teaches nothing. Keep every error, keep a small random sample of successes, and keep a trace id in the response so a user report can be matched to a run without asking the user for a timestamp. The trace id should be opaque, because it is also a handle into your system.

Redaction is part of the design, not a later pass. Email addresses, card numbers and document text should be removed or hashed before the trace is stored, and the rule belongs next to the handler so it cannot be forgotten when a new field is added. A trace store is a second copy of your data, with its own retention and access rules. Retention is part of the same decision: a store that keeps everything forever is a liability with a search box.