Measure, don’t vibe

The demo problem

Every LLM application works in the demo. You type three questions, the answers look good, and you ship. A week later someone reports a case that fails, you change the prompt to fix it, and three cases that used to work quietly stop working.

Nothing about that is mysterious. There was never a record of what “working” meant, so the second change had nothing to be measured against. The first three questions were evidence, but they were never written down.

The numbers make it concrete. Suppose the demo had five good answers. The fix repairs one reported case and breaks three of the five, and the next demo — three new questions — still looks fine. The regression is invisible not because the model is random, but because nobody kept the old cases in front of it.

A regression set is a list of cases

An evaluation set is a list of inputs with the output you would accept. Some cases have one exact answer; most have a property the answer must have. “Mentions the five-day window”, “cites a source”, “refuses because the document does not say”.

Once the list exists, running it is mechanical: send each case, score the reply, print a table. The value is not the score itself but the comparison — this change moved 42 of 50 to 46 of 50, and these four different cases now fail.

A case is small enough to write in a minute: an id, the input, the expectation, and a tag or two for the feature it covers. Stored as one JSON object per line, the set is diffable in review and readable without a tool. That matters more than it sounds — a case nobody can read is a case nobody will maintain.

The shape of a harness

cases = load_cases("evals/refunds.jsonl")

results = []
for case in cases:
    reply = chain.invoke(case["input"])
    results.append({
        "id": case["id"],
        "passed": check(reply, case["expect"]),
        "reply": reply,
    })

print(f"{sum(r['passed'] for r in results)}/{len(results)} passed")

Score properties, not prose

Comparing full text is brittle: two good answers rarely share a wording. What you can check reliably is narrower and more useful — does it contain the number, does it cite, does it refuse, does it avoid a forbidden claim.

A case can carry several checks, and it usually should. “Correct and grounded and short enough” is three properties, and a reply that satisfies two of them is a different problem from one that satisfies none.

A worked case: the reply “Refunds take five business days; support replies in two” against a case that requires the refund window and a citation. It passes the fact, fails the citation, and says nothing forbidden. Two of three properties, and the failing one names what to fix. A boolean would have said only “fail”.

The score is not the point

The number is a summary, and a summary is only useful next to the run it is compared with. “0.86” says nothing on its own. “0.86, up from 0.81, with two new failures in the upload cases” says what changed and where to look.

That is why the harness should print the per-case diff, not only the total. A change that adds three passes and loses one is a different decision from a change that adds two and loses none, even when the totals land in the same place.

Start with twenty

A hundred cases feels rigorous and is usually a way to postpone starting. Twenty real questions, honestly labelled, will find most of the failures in a young system, and they take an afternoon.

Grow the set from production. Every reported bad answer becomes a case; every case that no longer fails stays in the set as a guard. After a few months the list is the institutional memory of everything that has gone wrong.

Production is the best source because it supplies the cases nobody imagines. Give every reported bad answer an id, reproduce it once, and add it before the fix. A monthly pass that deduplicates and retires stale cases keeps the set from becoming an archive of features that no longer exist.