Researchers Propose Alternatives to Word-Error Rate

Asserting that traditional word-error rate (WER) calculations for speech recognition accuracy misses the mark, a team of speech researchers from European universities have proposed a pair of metrics that dive not just into how many words were wrong, but how much meaning was lost. I distinguishes semantically critical errors from harmless ones, noting that errors in words and phrases like industry-specific terminology, drug names, dollar amounts, legal instructions, or proper nouns could have very damaging impacts.

The researchers, Thibault Bañeras-Roux, Mickaël Rouvier, Jane Wottawa, and Richard Dufour, have proposed two measures related to morpho-syntactic and semantic aspects of transcribed words. The first, called POSER (Part-of-speech Error Rate), tracks whether transcription mistakes fall disproportionately on grammatically significant word classes, and EmbER (Embedding Error Rate), provides a weighting according to the semantic distance of the wrongly transcribed words

According to the researchers, standard WER benchmarks count substitutions, deletions, and insertions in transcripts and divide them by the total word count. The main flaw is that every word in a sentence is given the same weight though the value of some words measured against the downstream meaning of the full sentence is far greater.

"Evaluating automatic speech recognition (ASR) systems is a classical but difficult and still open problem, which often boils down to focusing only on the word error rate (WER). However, this metric suffers from many limitations and does not allow an in-depth analysis of automatic transcription errors," they wrote in the research summary.

They also asserted that applying language model rescoring changes the linguistic profile of errors in ways WER cannot detect, uncovering improvements and regressions in morpho-syntactic and semantic accuracy that WER scores did not.