Trusting the numbers

A score is an estimate

Running twenty cases and passing fifteen gives a score of 0.75. Run a different twenty and it might be 0.70 or 0.80 — the number describes the sample as much as the system. Reporting it without an interval invites decisions the data cannot support.

A confidence interval says how much the score would move if you sampled again. It is wide for small sets and narrow for large ones, and it is the honest way to present “seventeen of twenty”.

A worked interval makes the width concrete. Fifteen of twenty is 0.75, and a 95 percent Wilson interval runs roughly from 0.53 to 0.89. Seventeen of twenty is 0.85, with an interval from about 0.64 to 0.95. The two intervals overlap heavily, which is the arithmetic saying that the difference between them is not established by twenty cases.

Two runs, one comparison

The important question is rarely “what is the score?” but “is B better than A?”. Two point estimates cannot answer it. If A is 0.75 and B is 0.80, the difference is meaningful only if it is larger than the noise in both.

When both systems ran on the same cases, compare them case by case: B wins here, A wins there, both pass. The count of disagreements is the evidence, and it is far more sensitive than comparing two averages.

A worked paired result: A passes 15 of 20, B passes 16 of 20, and the totals look like a one-case gain. Case by case, B wins three and loses two, with fifteen ties. The evidence for B is three discordant cases in its favour, which is weak. Reporting only the totals makes it look like a clean improvement. The number of disagreements, not the gap in averages, sets the strength of the claim.

An interval in one function

import math

def wilson(successes, total, z=1.96):
    p = successes / total
    denom = 1 + z**2 / total
    centre = (p + z**2 / (2 * total)) / denom
    margin = z * math.sqrt(p * (1 - p) / total + z**2 / (4 * total**2)) / denom
    return centre - margin, centre + margin

print(wilson(17, 20))   # roughly (0.62, 0.86)

Flakiness is not noise

A test that passes and fails on the same input is a different problem. Temperature above zero, a live model that changed, a retrieval index that was updated — any of these makes the score move without the system changing.

Pin what you can: zero temperature for judged tasks, a recorded snapshot of the model output when testing downstream stages, and a fixed corpus for retrieval. What remains is genuine sampling variation, and that is what intervals describe.

Flakiness has a short list of causes. Temperature above zero, a model alias that points at a new snapshot, an index that refreshes mid-run, a tool with a timeout. Each makes the same input produce a different output, and each is fixable by pinning: zero temperature, a dated model id, a frozen corpus, a recorded tool response. Record a flaky case; the instability is information.

Report the denominator

A percentage without a denominator is a rumour. “92 percent” over twenty-five cases is twenty-three passes; over five hundred it is four hundred and sixty. The first could move ten points on one case. Always print n beside the rate, and print the interval beside both.

The same applies to slices. An overall rate can hide a feature that fails every time because it is a small part of the set. Report per tag when a tag has enough cases to say anything, and say “too few” when it does not.

What to report

Report the pass count, the total, the interval, and the list of failures. A dashboard that shows only “92 percent” hides the two things a reader needs: how much data it came from, and what broke.

For a release gate, compare against the previous run rather than an absolute number. “No case that passed before now fails” is a rule that catches regressions without pretending the absolute score is stable.

Put the point estimate last. The interval, the failures and the comparison are the parts a reader can act on; the estimate is the summary of them. A release note that says “no case that passed before now fails, and two new cases pass” is more useful than any single percentage.