Human review workflows

Humans are the instrument

Automated metrics are good at telling you that something changed and bad at telling you whether the new behaviour is acceptable. Human review is the other half: a person reads the output and applies a judgement the metric cannot encode. It is slower and more expensive, which is exactly why it should be aimed rather than sprayed at everything. A metric can tell you a regression happened; only a person can tell you whether the new answer is one you would stand behind.

Treat it as an instrument. A review queue with no question, no rubric and no sampling plan produces anecdotes with a spreadsheet attached. Decide first what decision the review will inform — ship or not, which failure to fix, whether a new intent is handled — and design the sample around that decision.

The output of a review is not a feeling. It is a label per item, a reason code, and a rate with an interval. Those three artefacts can be compared across weeks, which is what turns “the model seems worse” into a claim someone can check.

Sample in strata, not in bulk

Reviewing everything is rarely possible and usually wasteful. Stratify instead. Take all the failures from the automated checks, a sample of low-confidence outputs, a sample of a new intent, and a random slice of everything else. Each stratum answers a different question.

The failure sample tells you why things break; it says nothing about how often. The random slice is the only part that estimates the base rate, because every item had the same chance of selection. If you review only failures, your pass rate is zero by construction and your team learns nothing about the average.

Weight the strata back to the population when you report a single number. If a stratum is one tenth of traffic but half of your sample, its labels must count one fifth as much as they would in a naive average. Otherwise a deliberately enriched sample produces a rate that describes the queue, not the product.

Agreement, corrected for chance

from sklearn.metrics import cohen_kappa_score

kappa = cohen_kappa_score(reviewer_a, reviewer_b, labels=["pass", "fail"])
print(kappa)   # 0 is chance, 1 is perfect agreement

Calibrate before you count

A rubric is a contract between reviewers. It names the dimensions, the scale and what each point means, with an example at the boundary. “Quality 1 to 5” is not a rubric; “5 cites a supporting passage, 3 is correct but uncited, 1 is contradicted by the context” is.

Calibrate before counting. Have every reviewer label the same twenty items, compare, and discuss the disagreements. The goal is not consensus on every case but a shared reading of the rubric. If two reviewers cannot agree on what a 3 means, the number they produce is noise. Write the rubric before the queue opens and version it; a rubric changed mid-review makes the labels on either side incomparable.

Measure the agreement you achieved. Percent agreement is intuitive and flattering; Cohen’s kappa subtracts the agreement expected by chance, which matters when one label dominates. A kappa near zero means the rubric is not doing any work, however high the percentage looks.

Close the loop

Every disagreement is a signal. Either the rubric is ambiguous, the item is genuinely borderline, or one reviewer missed something. Adjudicate with a third reviewer, record the final label and the reason, and feed the reason back into the rubric or the example set.

Reviewer edits are data too. When someone rewrites a summary or fixes a citation, that diff is a candidate golden case and a description of the behaviour you want. A queue that discards the correction teaches the system nothing.

Watch the queue itself. Age, throughput and backlog decide whether the sample you planned is the sample you get. A review that arrives three weeks late cannot gate a release, and reviewers who are measured on speed will approve faster, not better. A team of two reviewers at forty items an hour has a real capacity limit; plan the sample around the people you have, not the people you wish you had.