Does Whisper Make Things Up? How to Spot It

Yes, sometimes. OpenAI's own model cards say Whisper's predictions "may include texts that are not actually spoken in the audio input", and that the models are prone to repetitive output. It does not happen on every recording and we cannot tell you how often, but a transcript that reads smoothly is not proof that it is right. The checks below find the common cases; the real protection is to listen to anything you will rely on.

What do the model cards say?

The Hugging Face model cards for openai/whisper-small and openai/whisper-large-v3 list the same cautions: hallucination (text not present in the audio), repetitive output that decoding settings reduce but do not eliminate, uneven performance across languages, and differences by accent and dialect. The small card also says capabilities such as speaker classification and diarization have not been robustly evaluated. We quote these as the authors' statements; we have not measured a hallucination rate and publish none.

Five checks that find invented text

1. Silence and music. Open the timestamped transcript. Any line whose timestamp falls in a stretch where nobody is speaking deserves a listen.
2. The same line again and again. A phrase repeated three or more times in a row is the loop the model cards describe. The script below finds them.
3. Fluent text that does not match. Play ten seconds and read along. A sentence that is grammatical but about something else is the dangerous kind.
4. Names, numbers and dates. They look plausible when wrong. Check each against the audio.
5. The wrong language. Text in a language the speaker did not use, or an unexpected translation, means the language setting or detection went wrong; choose the language by hand and rerun.

A script that flags repeated lines

import sys

# Flags a line that repeats three or more times in a row in a TXT or SRT transcript.
lines = [l.strip() for l in open(sys.argv[1], encoding="utf-8-sig") if l.strip()]
text = [l for l in lines if not l.isdigit() and "-->" not in l]   # drop SRT numbers and times
run = 1
for prev, cur in zip(text, text[1:]):
    run = run + 1 if cur == prev else 1
    if run == 3:
        print("repeated 3+ times:", cur)

Save it as repeats.py and run python repeats.py transcript.txt on a TXT or SRT download. It skips SRT numbers and time lines and prints any line repeated three times in a row. We ran it on a made-up file with a deliberate loop and on one without. A repeat is a hint to listen, not proof of an error: a speaker really can say the same word three times.

How to reduce it

• Skip silence. In the app's Advanced options, "Skip silence" is on by default; it keeps long silent stretches from being transcribed.
• Set the language. Detection reads the start of the file and can guess wrong on short or quiet clips.
• Trim long leading music or silence before you upload.
• If a loop starts, rerun that part on its own. Cut the passage into a short file and transcribe it again. We cannot promise this helps, but it is a cheap test.
• Prefer a clean recording. See how to improve transcription accuracy.

When this matters most

For anything quoted, published, legal, medical or financial, treat the transcript as a draft and have a person listen to the passages that matter. This is why every guide here calls a transcript a draft, and why we publish no accuracy percentage for your recordings. For how accuracy is measured, see word error rate: it counts inserted words too, so a transcript that invents text scores worse, but the number cannot tell you which of your lines are the invented ones.