Word Error Rate (WER): Formula and Example

Word error rate (WER) is the number of words a transcript got wrong, divided by the number of words in the correct reference text. Wrong means substituted, deleted or inserted, so WER = (S + D + I) / N, where N is the reference word count. Lower is better, 0% is a perfect match, and the figure can exceed 100% when a system inserts many extra words.

What do S, D and I mean?

• Substitution (S): a reference word replaced by a different word, for example "sat" written as "sit".
• Deletion (D): a reference word that is missing from the transcript.
• Insertion (I): a word in the transcript that was not in the reference.

The reference is a human-checked transcript of the same audio. The score comes from aligning the two texts so the fewest edits are needed, which is the word-level edit (Levenshtein) distance, then dividing by N.

A worked example

Two edits fix the transcript: "sat" became "sit" (one substitution) and one "the" is missing (one deletion), so S = 1, D = 1, I = 0. WER = (1 + 1 + 0) / 6 = 0.333, or 33.3%. A transcript can also score above 100%: against the reference "the cat sat", the output "the big black cat sat on a mat" adds five words (big, black, on, a, mat), so I = 5 and WER = 5 / 3 = 167%.

The same calculation in code

This is the whole algorithm. We ran it on the example above and it prints 0.333 (rounded):

def wer(reference: str, hypothesis: str) -> float:
    ref, hyp = reference.split(), hypothesis.split()
    # d[i][j] = fewest edits to turn the first i reference words into the first j hypothesis words
    d = [[0] * (len(hyp) + 1) for _ in range(len(ref) + 1)]
    for i in range(len(ref) + 1):
        d[i][0] = i                      # i deletions
    for j in range(len(hyp) + 1):
        d[0][j] = j                      # j insertions
    for i in range(1, len(ref) + 1):
        for j in range(1, len(hyp) + 1):
            same = 0 if ref[i - 1] == hyp[j - 1] else 1   # 1 = substitution
            d[i][j] = min(d[i - 1][j] + 1, d[i][j - 1] + 1, d[i - 1][j - 1] + same)
    return d[-1][-1] / len(ref)

print(wer("the cat sat on the mat", "the cat sit on mat"))   # 0.3333333333333333

A real example from our own measurement

On our measurements page, one Arabic clip (84 seconds of read speech) has 135 reference words; the Whisper small model made 22 errors on it, which is 22 / 135 = 16.3%. That is one clip. The page reports the pooled figure for three clips and explains how the text was normalised first.

What a WER number leaves out

• Normalisation changes the score. Whether you ignore capitals and punctuation, spell out numbers, or unify Arabic letter forms (alef, yeh, teh marbuta) before comparing can move WER a lot. Two scores are only comparable if the same normalisation was used.
• It counts every word equally. A wrong "the" and a wrong name or number each cost one error, though the name or number matters far more.
• It depends on the test set. Clean read speech, phone calls and meetings with overlapping speakers give very different scores for the same system.
• The reference can be wrong. The score is only as good as the human transcript it is measured against.
• It is not for every language. For languages written without spaces between words, results are often reported as character error rate instead; the Whisper README says its per-language figure uses "word or character error rates".

Is a WER number a promise about my recording?

No. A WER is a measurement on a particular set of recordings under particular rules. Your audio will have its own noise, accents, vocabulary and overlap. The fair way to compare tools is to take one minute of your own audio, write the correct text yourself, run each tool on it, and compute WER with the same normalisation for all of them. For how it relates to everyday accuracy, see how accurate AI transcription is.