Word error rate

The edit distance

Word error rate starts with an alignment. Line the reference and the hypothesis up and mark each position as correct, a substitution, a deletion or an insertion. The minimum number of edits is the Levenshtein distance (the smallest number of changes that turns one string into the other). It is computed by a table where each cell is the best cost for a prefix. The recurrence (the rule that fills each cell from its neighbours) is three cases, and the backtrace recovers the actual mistakes rather than only their count. That backtrace is what makes an error report useful.

The four operations are not equivalent, even though the table charges one for each. A substitution changes a word, a deletion loses one, and an insertion adds one the speaker never said. For readability, the alignment is usually printed with the reference on top and the hypothesis below, and the operations between them. A human can scan ten lines and see whether the system is dropping endings or confusing two similar names. The cost of each operation is a modelling choice, not a law.

From distance to rate

The rate is the total edits divided by the number of reference words. Substitutions, deletions and insertions all count once. A perfect transcript scores zero. A transcript that inserts a word for every reference word scores one hundred percent, even though every word is wrong in a different way. Because insertions are unbounded, the rate can exceed one hundred percent, and that is not a bug in the formula. It is a distance normalised by the reference length, not a probability. A rate above one is a signal that the system is inventing words.

Normalisation matters as much as the distance. Case, punctuation, contractions and filler words are usually stripped or standardised first, and the choice changes the number. “Umm” removed from both sides can turn a five percent error into a two percent error without changing the system. Write the normalisation down with the score, because a WER without its preprocessing cannot be compared to any other WER. The same applies to numbers and spelled-out units.

A WER with an alignment report

import jiwer

reference = "the quick brown fox"
hypothesis = "the quik brown box"

print(round(jiwer.wer(reference, hypothesis), 3))
print(jiwer.process_words(reference, hypothesis))

Corpus rate and utterance rate

There are two averages and they answer different questions. The corpus rate sums all edits and divides by all reference words, so long utterances dominate. The mean of per-utterance rates weights a two-word command as much as a fifty-word sentence. Report the corpus rate when you care about total transcription work, and the per-utterance table when you care about consistency. A single number is always a summary of a distribution. Show the distribution at least once.

A corpus rate can hide a catastrophic (disastrous) subset. If the system is perfect on read speech and unusable on accented calls, the average may still look acceptable. Slice the results by speaker, accent, channel, duration and noise level, and report the worst slice. That is where the next piece of work is, and it is also where a deployment decision should be made. An average is a claim about the whole, not a promise about any part. A slice with ten examples is anecdote; a slice with a thousand is evidence.

What the number hides

WER treats all words as equally important, which is false. Getting a name wrong can cost more than getting an article wrong, and a substitution that keeps the meaning still counts. It also ignores word order beyond the alignment and gives no credit when a word sounds the same. For read speech with a clean reference it is a good, cheap metric. For dialogue, domain terms or anything a user will act on, it needs company.

The alternatives are not replacements so much as different lenses. Character error rate is gentler for languages with rich word forms and harsher for short words. Keyword error rate focuses on the terms that matter. Semantic metrics compare meaning rather than surface form. Human review remains the ceiling, and even humans disagree on hard audio. Use WER as the stable baseline, then add the metric that matches the decision the number will support.