Capstone: an eval harness

Four moving parts

The loader reads cases from a file. The runner calls the system under test. The checker applies the case’s expectations. The reporter prints a row per case and a summary, and returns a non-zero exit code when the gate fails.

Those four parts fit in one file and stay useful for years. Everything else — parallelism, dashboards, model judges — is an addition to one of them, not a replacement.

A run looks like this. The loader reads twelve cases. The runner sends each input and gets a reply. The checker returns a list of failed checks per case, usually empty. The reporter prints eleven passes, one failure with its reason, and exits one. The exit code is the part CI reads; the row is the part a person reads.

The gate is a comparison

An absolute threshold invites arguments about whether 0.86 is good. A comparison is unambiguous: this run has three cases that the baseline passed and this one does not, and here they are.

Store the baseline next to the cases. evals/baseline.json is a list of case ids that passed on the last accepted run; the gate fails on any id that disappears. That is a regression test in the ordinary sense.

A worked gate: the baseline has ten passing ids, and the candidate passes nine of them but also passes a new case. The pass count is unchanged at ten, and the gate still fails, because upload-limit used to pass and no longer does. That is the point — a regression is a lost case, not a lower average.

The comparison also shows what not to do. If the candidate passes a case the baseline failed, that is an improvement and worth reporting, but it does not cancel the regression. Two lists, gains and losses, keep the decision honest.

Running it in CI

# .github/workflows/evals.yml
- run: python evals/run.py --baseline evals/baseline.json
# exit code 1 on a regression, which fails the job

Keep the failures readable

A failure message should name the case, the check and enough of the reply to see what happened. “3 cases failed” costs another run to investigate; “refund-window: missing five business days — reply was: …” does not.

The same applies to the summary. Print the count, the interval, and the list of failing ids. Someone reading a CI log at midnight should not have to open a dashboard.

A good message reads like a sentence: “refund-window: missing ‘five business days’ — reply was: ‘Refunds take about a week.’” It names the case, the check and the evidence. The next person can decide whether the system is wrong or the case is, without re-running anything.

Keep the harness fast

A harness that takes twenty minutes is one people stop running. Twenty cases at two seconds each is forty seconds, fine for every pull request; two hundred is seven minutes, which is not. Split the set: a smoke slice on every change, the full set nightly.

The same arithmetic applies to the system. If each case calls a live model, the harness measures the network as much as the answer. Record the output for downstream cases, and keep a small live slice.

What comes after

With a harness in place, the improvements become ordinary engineering: add a case for every production failure, split the set when it grows past a minute, and add a judge only for the properties that a rule cannot express.

The habit is the point. A team that measures before and after every prompt change will beat a team that argues about prompts, not because its model is better but because it knows which changes were real.

The baseline is a reviewed file, not an artefact the harness rewrites. When a case legitimately changes, update the baseline in the same pull request and say why. A gate that updates itself is a gate that always passes.

Give the harness an owner. Someone has to decide when a flaky case is quarantined, when a slow suite is split, and when a baseline change is honest. Without that, the gate survives until the first inconvenient failure.