The Proper Noun Problem in Speech Recognition
I found the problem while dictating a piece about northern Japan and the North Pacific. The transcript looked fluent. It was also wrong where it mattered. Ainu became I knew. Jomon was mangled. Ryukyuan became Ryugu, Yamato became Amado, and Sakhalin turned into saccline.
None of those errors made the sentence collapse grammatically. They damaged the names.
That distinction matters for anyone evaluating automatic speech recognition. Word error rate, or WER, remains indispensable because it gives us a common measure of insertions, deletions, and substitutions. But it is an average. It does not know that one substitution changed a filler word while another changed a person, place, drug, product, treaty, company, or technical term.
A transcript can therefore look excellent in aggregate and still be operationally bad.
NIST has already treated this as a distinct evaluation problem. In its OpenASR21 challenge, it introduced a case-sensitive scoring track for three languages as a proxy for proper-noun performance. The point was not that capitalization solves names. It was that conventional scoring can miss a class of errors whose consequences are disproportionate to their frequency.
The newest specialized-vocabulary research makes the problem even clearer. A 2026 LREC paper split errors into biased WER for domain jargon and unbiased WER for ordinary vocabulary. On the hardest test sets, jargon error rates reached 0.88 to 0.90 while ordinary-vocabulary error rates were only 0.06 to 0.19. A single overall WER would blur that gap.
So why do names and rare terms fail so easily? Frequency is one part of it. Common words appear repeatedly in training data; a surname, indigenous nation, specialist component, or local place name might appear rarely or not at all. Phonetic ambiguity makes the problem worse: when the audio could plausibly map to a familiar phrase, the recognizer has strong statistical reasons to prefer what it has seen before. Modern subword tokenization can also make an unfamiliar name an awkward sequence of pieces rather than a well-supported lexical unit.
The practical consequence is that sounds fluent is not a reliability test.
Organizations buying or deploying speech recognition should build a proper-noun challenge set before they compare systems. The set should come from the actual deployment environment: customer names, local geography, product families, staff surnames, medical or legal terminology, acronyms, indigenous names, multilingual entities, and any words whose corruption would break search, routing, compliance, identity, or downstream analytics.
Then score those terms separately.
A useful evaluation stack has at least four layers. Keep overall WER for comparability. Add an entity-specific or biased WER for the terms you care about. Measure entity recall: how often did the system actually recover the target name? And add a severity review for consequential errors. Tuesday becoming Thursday in a meeting transcript might matter more than several harmless article substitutions; a patient, claimant, or customer name might matter more still.
The test should also be adversarial in a mundane way. Vary accents, microphones, speaking rate, background noise, code-switching, and sentence position. Include names that resemble common words. Include names the language model is tempted to normalize into something more familiar. Do not let the benchmark consist only of terms the vendor already knows.
There are technical mitigations. Contextual biasing can supply a list of likely entities at inference time. Vocabulary injection can prioritize domain terms. A 2024 LREC paper on CB-Whisper showed that contextual biasing can substantially improve entity recall, and newer work explores dynamic vocabularies that add bias terms without rebuilding the whole recognizer. The 2026 specialized-vocabulary benchmark also found that providing the correct jargon through prompting could cut biased WER dramatically.
But biasing is not magic. A large or poorly chosen vocabulary can create new confusions. A system can over-favor the supplied term when the speaker actually said something else. So the question is not merely whether a platform has a custom vocabulary feature. The question is whether the organization can measure what that feature fixes, what it breaks, and under which acoustic conditions.
For consequential names, there should also be a verification path. Flag low-confidence or high-value entities for human review. Preserve the original audio where policy permits. Let users correct names once and reuse those corrections where appropriate. If downstream systems depend on entity extraction, test the end-to-end chain rather than assuming a plausible transcript is enough.
The larger lesson is simple: speech recognition does not fail uniformly. If an ASR system gets 95 percent of ordinary words right and repeatedly misses the 5 percent that identify people, places, products, and obligations, the business problem is not 5 percent error. It is concentrated semantic failure.
The proper-noun problem is, therefore, not a niche linguistic annoyance. It is an evaluation-design problem. Once buyers and developers measure the important words separately, they can stop asking only, How accurate is this transcript? and start asking the more useful question: Accurate on what?
Joshua W.J. Brown is a Toronto-based writer, filmmaker and independent researcher whose work focuses on technology, accessibility, evidence, archives and public systems.
Process disclosure: OpenAI ChatGPT was used for research retrieval and comparison and for drafting/refinement. Brown verified the cited source propositions, made the final editorial decisions and accepts responsibility for the submitted text.