AI Video Transcription: How It Works
What AI transcription actually does, why it is so much better than it was, and the specific situations where it is still unreliable.
The Scriptorium team ·
AI video transcription means using a machine learning model to turn the speech in a video into text. It got dramatically better around 2022 and has been good enough for everyday use since. It is also still wrong in specific, predictable ways, and knowing which ones is the difference between a useful tool and a liability.
How it works
A speech recognition model is trained on a very large collection of audio paired with its correct transcription. From that it learns to map sound to text. Given a new recording it predicts the most likely sequence of words.
The important word is likely. Modern models consider surrounding context rather than each sound in isolation, which is why they punctuate sensibly, capitalise names, and usually pick the right homophone. It is also why they fail the way they do.
Why it improved so much
Two things changed. Models moved to architectures that use much longer context, so a word is interpreted in light of the whole sentence around it rather than the syllable before it. And training data grew by orders of magnitude, covering many languages, accents and recording conditions instead of clean studio speech.
The practical result is that transcription went from something that needed per-speaker training and still produced comedy, to something you can point at an arbitrary recording and expect usable text from.
Where it still fails
Overlapping speech
Two people talking at once is the hardest problem in the field. A model trained to produce one stream of words has no good answer when there are two. Expect dropped words, merged sentences, or one speaker disappearing entirely.
Proper nouns
A model cannot know how your colleague spells their name, what your product is called, or which of four similarly-pronounced towns you mean. It guesses from context, and context is often not enough.
Specialist vocabulary
Medical, legal and technical terms are underrepresented in training data relative to how often they appear in the recordings people actually want transcribed.
Heavy accents and code-switching
Accuracy varies by accent, and it drops when a speaker switches between languages mid-sentence, which is completely normal in much of the world and poorly handled almost everywhere.
Confident wrongness
The one to internalise. Poor audio does not produce obvious garbage. It produces clean, grammatical, confident sentences that are simply not what was said. A transcript that reads well is not evidence that it is correct.
What that means in practice
- Treat the output as a first draft, not a record.
- Check it against the audio, not on its own.
- Check names, numbers and dates specifically.
- Never quote from an unchecked transcript.
- For legal, medical or publishable work, have a human verify it.
Does it learn from my recordings?
It depends on the service, and it is a fair question to ask each one directly. Recognition models are trained in advance; a transcription service does not need to train on your audio to transcribe it. Whether it keeps and uses your files for anything else is a policy decision, not a technical necessity.
For this service, uploaded files are transcribed and stored for you, and are not used to train models. Files from the free no-account tier are deleted after 24 hours. See the privacy page.
Trying it
The fastest way to judge whether it is good enough for your work is to run your own hardest recording through it — not a clean one. transcribe video takes a file directly, free for a first file up to 15 minutes.