Evaluating agents

The answer is only half the output

An agent produces two things: an answer and a path to it. The answer can be scored like any other output. The path — the tools it called, the arguments, the steps it took and the way it stopped — is the part a final-answer metric cannot see, and it is where most of the interesting failures live.

A right answer reached by guessing is a latent failure: the next input will not be so lucky. A wrong answer reached by a sensible path is often a retrieval or data problem rather than a reasoning one. Scoring only the answer collapses those two into the same red cell.

Trajectory evaluation separates them. You record every step, then ask questions about the record: did it search before answering, did it call the refund tool, did it stop inside the budget? Those are cheap checks and they explain a failure in a way a score never does.

Check the path with rules, not exact match

The obvious metric is exact match against a reference trajectory, and it is almost always the wrong one. Two correct paths can differ in the order of two independent lookups, or in whether the agent retried a failed call. Exact match fails both and rewards a memorised sequence.

Check properties instead. A required set — the tools that must appear somewhere. A forbidden set — the ones that must not. An order constraint — search before answer, draft before send. A budget — no more than six steps, no more than three calls to the paid API. Each is a rule you can explain to a reviewer.

Rules are not a judge and do not need to be. They catch the failures you already know about, cheaply, on every run. A model judge is for the residue: whether the path was reasonable, whether the agent gave up too early, whether a tool was chosen for a defensible reason.

A trajectory evaluator

from langsmith import evaluate

def trajectory_ok(run, example):
    calls = [step["name"] for step in run.outputs["trajectory"]]
    required = set(example.outputs["required_tools"])
    return {"key": "required_tools", "score": required <= set(calls)}

evaluate(agent, data="agent-cases", evaluators=[trajectory_ok])

Run it more than once

An agent is nondeterministic even at temperature zero, because the tools are. A search returns a different top result, an API times out, a model samples a different tool. One run tells you what happened once, not what the system does.

Run each case k times and report two numbers. pass@k is the chance that at least one run succeeds — useful for a task where a retry is acceptable. pass^k is the chance that all k succeed — the number that matters when a user sees every attempt and a failure has a cost.

The gap between them is consistency. A system with pass@5 of 0.95 and pass^5 of 0.55 succeeds eventually and fails often, which is exactly how an agent feels unreliable in production even when the demo works. Report the distribution of steps and cost too: a median of three steps with a tail of twenty is a different product from a flat four.

Count the failure modes

A single pass rate hides which failure you have. Count categories instead: wrong tool, wrong arguments, loop, gave up, claimed success without acting, and unsafe action. Each maps to a different fix — a tool description, a schema, a budget, a prompt, a verification step or a permission. Keep the taxonomy small enough that two people apply it the same way: six categories, not sixty.

Fixtures make the count stable. Replace every live tool with a deterministic fake that returns a recorded response, and record the calls it received. The evaluation then measures the agent rather than the weather, and a failure can be replayed exactly.

Keep one live smoke test outside the scored set. Fixtures tell you the logic is right; a live run tells you the integration still exists, and neither substitutes for the other. A fixture that never fails tests the happy path and nothing else, so add a timeout and a malformed response to the set.