Exact match and token F1
The two oldest metrics in question answering, and what each one is blind to. Exact match is unforgiving; token F1 gives partial credit and can be gamed by repeating words.
Exact match, with a fair chance
Exact match compares the expected answer with the reply after normalisation: lowercase, strip punctuation, collapse whitespace, drop articles. Without that step the metric measures formatting rather than correctness, and “Five business days.” fails against “five business days”.
It is the right metric when the answer is a fact with one spelling — a date, a code, a number. It is the wrong metric for a sentence, because two correct sentences are almost never identical.
The normalisation step is where most of the argument lives. “Five business days.” and “ FIVE business days ” should both pass; “within five business days” should not, because the expectation is the bare fact. If extra words are acceptable, the expectation should be a property rather than an exact string.
Token F1 gives partial credit
Token F1 treats the answer as a bag of words. Precision is the share of the reply’s tokens that appear in the expected answer; recall is the share of the expected tokens that appear in the reply; F1 is their harmonic mean.
It rewards saying the right things and penalises both noise and omission. A reply that adds a paragraph of context scores lower precision; one that omits the number scores lower recall. That is usually the behaviour you want from a short-answer metric.
A worked score makes the trade visible. Expected: “refunds are issued within five business days”, seven tokens. Reply: “refunds are issued within five business days for card payments”, ten tokens, with all seven expected tokens present. Precision is seven tenths, recall is seven sevenths, and F1 is 0.82. The reply is correct and still loses points for the extra phrase, which is the metric doing what it was asked to do.
The same comparison in a library
from evaluate import load
exact = load("exact_match")
f1 = load("squad_v2")
print(exact.compute(predictions=[reply], references=[expected]))
print(f1.compute(predictions=[reply], references=[expected]))Normalise before comparing
Run the comparison, then add a reply that is correct but capitalised and punctuated differently. Without normalisation it fails; with it, it passes.
What F1 cannot see
A bag of words has no order and no structure. “The refund takes five days, not thirty” and “The refund takes thirty days, not five” share every token and score identically. Negation, comparison and ordering are invisible.
It can also be gamed. Repeating the expected words in any order raises recall; padding with related terms raises precision. The metric is a useful signal and a poor judge, which is why it is paired with a model-based check for open-ended answers.
A worked contradiction: “The refund takes five days, not thirty” and “The refund takes thirty days, not five” contain the same tokens in a different order, so F1 is 1.0 for both. One is right and one is wrong. If that distinction matters, the check has to look at order or negation, and a bag-of-words metric cannot.
Report exact match and F1 together. A large gap between them says the answers are semantically right but worded differently — useful information about the prompt, not the model.
Pick the metric from the answer shape
The answer shape decides the metric. A single fact with one spelling — a date, a status, an invoice number — is exact match after normalisation. A short phrase with acceptable paraphrases is token F1, with the caveat that order is invisible. A sentence, a summary or a multi-step explanation is property checks for the facts that must appear, plus a judge or a person for the rest.
Using the wrong shape is not a small error. Exact match on a summary scores near zero for correct answers, and token F1 on a long answer rewards length. Both move for reasons unrelated to quality.
When to stop using them
As soon as the expected output is longer than a phrase, token overlap stops tracking correctness. Summaries, explanations and multi-step answers need a different measure: property checks for the facts that must appear, and a judge or a human for the rest.
Keeping a token metric on long answers is worse than dropping it, because it produces a number that moves for reasons unrelated to quality and invites tuning against the wrong thing.
The symptom is easy to spot: the token score improves when answers get longer, or when the model repeats the question. That is not correctness; it is the metric responding to the shape of the text. Drop it before it becomes the number someone optimises.
Precision, recall and F1
Score three replies against the expected answer. Then write a reply that repeats the expected words in the wrong order and see that F1 cannot tell.
Check what you learned
One question at a time. A wrong answer explains itself — take another go.
Part 1 of 5
Why normalise before an exact-match check?