The golden dataset

Cases come from real questions

The best source of evaluation cases is the questions people actually asked, including the badly spelled ones and the ones your system answered wrong. Support tickets, search logs and chat transcripts are all richer than anything invented at a desk.

When there is no production traffic yet, generate candidates with a model and then edit them by hand. Synthetic cases are fine as a starting point and dangerous as a finish line: they inherit the model’s idea of a good question, which is rarely the user’s.

A real ticket shows why. “hi i was charged twice for the team plan can i get one back” contains a misspelling, two intents and no punctuation, and the answer depends on both the billing policy and the refund window. A model asked to invent a billing question writes something clean and single-purpose. The messy version is the one that finds the bug.

Coverage beats size

A thousand questions about refunds measure one thing well. Twenty questions spread across billing, support, uploads, permissions and refusals measure the product. What matters is that every behaviour you claim to support appears at least once.

Draw the matrix: rows are features, columns are case types — typical, edge, adversarial, unanswerable. Every cell needs a case. Empty cells are the bugs you have not met yet.

A worked matrix has three features and four kinds, so twelve cells. The empty ones are rarely the obvious: billing has typical, edge and adversarial cases but no unanswerable one, which means nobody has checked that the system refuses a question about a refund it cannot see. That is the cell where a confident guess hides.

One case per line

{"id": "refund-typical", "input": "How long do refunds take?",
 "expect": {"must_contain": ["five business days"], "must_cite": true},
 "tags": ["billing", "typical"]}

{"id": "refund-unanswerable", "input": "Can I refund a gift card?",
 "expect": {"must_refuse": true}, "tags": ["billing", "unanswerable"]}

Label with a rule, not a mood

Two people labelling the same case should reach the same answer. That requires a written rule: what counts as grounded, what counts as a refusal, how much detail an answer needs. Without it, the labels drift and the score measures the labeller.

Where a rule cannot be written, the case is probably too subjective to be a pass/fail check. Keep it as a review example instead of pretending it is a metric.

The rule can be short and still decide cases. Grounded means every number and date in the answer appears in a cited passage. A refusal means the answer says the documents do not cover the question and names no fact. A partial answer that gives one correct fact and one invented one is not grounded, and the label is fail, not partial. Write those three sentences down and two labellers will agree far more often.

Three slices, not one

A working set is usually split three ways. The development slice is what you look at while changing prompts. The regression slice is the growing list of past failures, run on every change. The held-out slice is run rarely, and it is the only honest estimate of whether the improvement generalises.

The proportions matter less than the discipline. If the held-out slice is opened every day, it becomes development data with a different name. Put it behind a flag or a scheduled job so that opening it is a decision rather than a habit.

Keep the set alive

A dataset is a living artefact. Add every production failure, retire cases whose feature was removed, and review the labels when the product’s behaviour changes. A stale set silently passes a system that no longer does what it promises.

Version it with the code. A score is only meaningful next to the set it was measured on, and “it was 92 percent last month” means nothing if the cases changed underneath.

Give the set an owner as well as a version. Someone has to decide that a case is stale, that a label is wrong, or that a feature no longer exists. Without an owner, the set grows by addition only, and its failures slowly stop corresponding to anything the product does.