How to Label Speakers in a Transcript
How automatic speaker labelling actually works, why it sometimes miscounts or swaps speakers, and how to get labels you can trust once a transcript is back.
The Scriptorium team ·
Automatic speaker labelling groups segments of a transcript by how similar the voices sound, without being told in advance how many people are on the recording. That is useful and imperfect in ways worth understanding before you rely on a label to say who said what.
What it is actually doing
The process compares short stretches of audio and clusters the ones that sound like the same voice, using a distance threshold rather than a fixed speaker count. It has no idea who "Speaker 1" is — it only knows that a set of segments sound more like each other than like the rest. Turning "Speaker 1" into a real name is a step you do afterwards, using context from the conversation.
Why the count is sometimes wrong
- Two people with similar-sounding voices can be merged into one label.
- One person moving further from the microphone, or a phone changing hands, can shift a voice enough to be split into a second label.
- Cross-talk — two people speaking at once — has to be assigned to somebody, and a clustering step over a short overlapping segment will sometimes assign it to the wrong one.
None of this means the transcript's words are wrong — the text is usually still accurate. What can be wrong is which label a correct sentence is attached to.
Getting labels you can rely on
- Treat the automatic labels as "voice A" and "voice B" rather than as verified identities until you have checked a few lines of each against the audio.
- Rename labels to real names once you are confident which is which — most people do this from a line near the start where someone is introduced or introduces themselves.
- Where a label matters for something that will be quoted or acted on, confirm that specific line against the audio rather than trusting the label alone.
Recordings with clearly distinct voices, minimal cross-talk and each speaker close to a microphone give the cleanest labels. A single phone in the middle of a busy room asks the most of automatic labelling and is where manual correction earns its keep.
When there is no automatic labelling at all
Speaker labelling depends on a speaker-embedding model being available; where it is not, a recording still transcribes normally and falls back to manual labelling — you assign names to segments yourself rather than starting from an automatic guess. The transcript is not degraded, only the labelling step.