Normalize Arabic Text for WER in Python

Before you compare an Arabic transcript with its reference, remove the diacritics and tatweel, unify the alef forms, map alef maksura to yeh and teh marbuta to heh, and strip punctuation; otherwise spelling variants that do not change the meaning count as word errors. This page gives the function our own benchmark uses, so you can reproduce the Arabic figures on our measurements page and compute comparable ones for your own recordings.

The function

import re

# Diacritics and marks: U+0610-061A, harakat U+064B-065F, superscript alef U+0670,
# Quranic annotation marks U+06D6-06ED, and tatweel U+0640
MARKS = re.compile("[\u0610-\u061a\u064b-\u065f\u0670\u06d6-\u06ed\u0640]")


def normalize_ar(text):
    text = MARKS.sub("", text)
    text = re.sub("[\u0622\u0623\u0625]", "\u0627", text)               # alef with madda or hamza -> bare alef
    text = text.replace("\u0649", "\u064a").replace("\u0629", "\u0647")  # alef maksura -> yeh, teh marbuta -> heh
    text = re.sub(r"[^\w\s]+", " ", text)                                  # punctuation -> space
    return text.split()

It is the same normalisation as in the benchmark script in our repository (we checked that the two give identical output on test strings). Combine it with the word-level edit distance in the word error rate guide: run normalize_ar on the reference and on the transcript, then compute the distance between the two word lists.

What each step does

Before and after

The table is the function's real output. The first rows show a variant spelling collapsing to a form that both a reference and a transcript would share, so the difference no longer counts as an error; the last row shows punctuation being dropped.

If you only need to strip the marks from some text, without code, the Arabic diacritics remover runs in your browser and has optional letter-form unification; check its options against the list above before comparing results.

The limits of this normalisation

• It can hide a real error. Mapping alef maksura to yeh makes على (on) and علي (Ali) the same string, and mapping teh marbuta to heh merges forms that differ in grammar. A strict score would keep them apart; this one does not.
• It does not unify everything. Hamza on waw or yeh (ؤ and ئ) is left as is, digits are not converted between Arabic-Indic and Western forms, and spacing differences inside a word are not repaired.
• It is a choice, not a standard. Whatever you pick, use the same function for every system you compare, and say which one you used. Two scores are comparable only under the same normalisation.
• It changes the number. In the Arabic guide's example, a missing word and a tanween make the error rate 40% without normalisation and 20% with it (the example in Arabic).

Where this fits

For how to run a fair comparison between tools on your own audio, see the ten-minute test, and for the Arabic word-count differences that affect the denominator, how many words an hour of audio is.