Remove Arabic diacritics (tashkeel) from text

What diacritics are, and why people remove them

Arabic is normally written without short vowels. The marks that show them, called tashkeel or harakat, appear in the Quran, in children's books, in dictionaries and in poetry, and are left out of almost everything else. Text that carries them looks different to a computer: the word كَتَبَ and the word كتب are not the same string, so a search for one will not find the other. Removing the marks is the usual first step before searching, comparing, counting or loading text into a tool that does not expect them.

The options

• Diacritics. On by default. Removes the vowel marks, the shadda that doubles a consonant, the sukun and the tanwin endings.
• Tatweel. The stretching line some writers insert for looks, as in العــــربية. It is not a letter and not a diacritic, and a search for the plain word will not match it.
• Quranic signs. The small annotation marks used in Quran text (U+06D6 to U+06ED).
• Digits. Switch between Western digits (0 1 2 3) and the Eastern Arabic-Indic ones (٠ ١ ٢ ٣). Persian digits are read as Eastern ones when converting to Western.
• Punctuation. Switch between the Arabic comma, semicolon and question mark (، ؛ ؟) and their Latin forms.

The letter options are for matching, not reading

Three more options rewrite letters: every alef with a hamza or madda becomes a plain alef, alef maqsura becomes yaa and taa marbuta becomes haa. They are deliberately separate and off by default, because each merges words that mean different things. They exist because people often type the letters loosely, so a search index works better when the loose and strict forms are made equal. Do not use them on text a person will read.

An example

The sample text, السَّلامُ عَلَيْكُمْ وَرَحْمَةُ اللهِ, becomes السلام عليكم ورحمة الله with the default option. The note under the result counts how many marks were removed, so you can see at once that it did something, and a result that says nothing changed tells you the text had no such characters to begin with.

Working with a file

You can paste text or choose a plain text file. The result is shown in a box you can copy from or download as a text file, and the file name is kept so a download of notes.txt is saved as notes.txt again. Very long texts work, since all the work is a handful of character replacements, but if you are cleaning a whole book it is faster to do it in a script with the same rules: the code points are listed above.

Where this fits with transcription

A transcript of speech is usually written without the vowel marks, so this tool matters most for text that comes from elsewhere: a book, a lesson plan, a poem you want to compare with a transcript. If you have the recording of an Arabic talk, Scriptorium turns the speech into text you can edit; see Arabic transcription for how Arabic is handled, and the SRT to text tool for taking the words out of a subtitle file before you clean them here.