Agent evaluation

You cannot eyeball a loop

A single model call has one output, so a test can compare it to an expected string. An agent has a trajectory: the tool calls it chose, the arguments it passed, the observations it received, the retries and the point where it stopped. Two runs can reach the same answer by very different paths, and only one of those paths is safe to ship. A run that reaches the answer in three steps is not the same product as one that reaches it in twelve.

That is why reading the final answer is not enough. A correct answer produced by reading a file the agent should never have touched is a security failure with a good ending. The trace is part of the output, and the evaluation has to look at it. A run that needed nine steps for a three-step task is also a finding, even when the answer is right. Record the trace on success too, because the next failure will be compared against it.

Three layers, cheapest first

Unit checks cover the deterministic parts: a tool schema rejects a missing argument, a parser handles an empty response, a policy refuses a forbidden action. These are cheap and fast, and they should run on every commit. They catch the failures that have nothing to do with the model, which are more common than people expect. They also run in milliseconds, which means nobody is tempted to skip them.

Trajectory checks compare what the agent did with what it was allowed to do: which tools it called, in what order, how many steps it took and whether it stayed inside the budget. End-to-end checks ask whether the task was completed at all, and they are the slowest and most expensive. Run them in that order, because one failure at the bottom explains a hundred failures at the top. The expensive layer should never be the first to tell you that a schema is broken.

An evaluation run in LangSmith

from langsmith import evaluate

def exact_match(run, example):
    return {
        "key": "correct",
        "score": run.outputs["answer"] == example.outputs["answer"],
    }

results = evaluate(
    lambda inputs: agent.invoke(inputs["question"]),
    data="agent-regression",
    evaluators=[exact_match, steps_within_budget],
)
print(results.experiment_name)

Graders and their biases

An exact match is the best grader when the answer has one right form, and a poor one when it has many. “Five business days” and “within 5 working days” are the same answer, and a string comparison calls one of them wrong. Normalise the text before comparing, or use a check that reads the claim rather than the wording. A normaliser that lowercases, strips punctuation and maps numbers to words fixes most of it.

A model as judge is useful for tone and completeness, but it needs a rubric with named criteria and a few scored examples. It also has known biases: it prefers the first option, longer answers and text that sounds confident. Calibrate it against human labels on a sample, and never let it be the only grader standing between a change and a release. Keep the judge prompt versioned like any other prompt, and re-calibrate when it changes.

Regression, not a leaderboard

The number to watch is the pass rate on a fixed suite, compared with the previous release. A leaderboard position is not a regression test. Slice the rate by category, because an overall score that stays flat can hide a collapse in one kind of task and an improvement in another. The category is usually where the interesting change lives. A category with three cases is a hint; a category with thirty is a measurement.

Cost and latency belong in the same report. An agent that improves the pass rate by taking twice as many steps may still be the wrong release. Record the variance too: a task that passes two times out of three is not a passing task, it is a flaky one, and flakiness usually means the agent is guessing rather than reasoning. Report the median and the worst case, not only the mean, because the worst case is what a user remembers.