What Word Error Rate Misses on Real Calls
Most teams choose a speech-to-text model the same way: They open a leaderboard, sort by word error rate (WER), and shortlist the top three. The number looks standard and easy to compare, and it seems to answer buyer's most pressing questiont: How often will this model mishear my customers?
It answers a narrower question: How often did this model mishear these particular recordings? When those recordings sound nothing like a customer call, the answer can point a buyer the wrong way.
Yi Zhong's September column pointed out that WER counts a dropped filler word and a wrong digit in an account number as one error each. That is a problem with what WER counts. WER has a second, easily missed problem: where it is counted.
The Same Models, Two Different Rankings
My company, Cekura, tests voice agents, and as part of that work we publish benchmarks of the components from which they are built. Our latest speech-to-text benchmark ran 19 streaming models on two sets of audio. The first was 1,000 short public clips from Pipecat's open benchmark dataset. The second was eight recordings from four real two-person phone conversations, licensed from Ocular AI, with each speaker on a separate track. That second set is not public.
On the clean clips, the most accurate model had a WER of 1.75 percent. On the phone conversations, the same model's error rate was 4.83 percent, almost three times as high, and the gap is well outside the margin of error on either set.
It was not an outlier. The five most accurate models on the clean clips all had at least double the error rate on the phone calls. Of the 19 models, 16 did worse on the conversations, and the median increase was 54 percent. The model with the lowest error rate on the conversations, at 2.93 percent, had ranked sixth on the clean clips.
Counted in words, the conversations contain 12,561 words. At 2.93 percent, a model gets about 370 of them wrong. At 4.83 percent, it gets about 610 wrong. A buyer who shortlisted the top three models on clean audio would have picked models that scored 4.5 to 5.2 percent on the conversations, against 2.93 percent for the leader. With only eight recordings, many of the real-call scores sit inside each other's margins, so the exact order there shouldn't be over-read. The jumps for the five clean-clip leaders, though, are well outside the margin of error.
Why Clean Scores Don't Travel
Clean clips and phone conversations differ in ways that matter to a recognizer. In conversation, people talk over each other, restart sentences, trail off and say "mm-hmm" while the other person is still talking. Callers are on speakerphones, earbuds and car systems. A short, clean clip has none of that.
Training data might also play a part. Speech models are trained on very large amounts of audio collected from the web; OpenAI trained its Whisper model on 680,000 hours of it. A test set that is published online can end up in a model's training data, and a model that has seen the test will score well for reasons that won't carry over to customer calls. Researchers call this test-set contamination. Our data can't say why any model moved as much as it did, and we are not suggesting contamination for any of them, but a held-out set of real conversations is one of the few ways to check.
Small Differences Are Often Noise
Leaderboards usually show one number per model, and a single number invites over-reading. On our real-call set, 14 other models fell within the margin of error of the leader. On the clean clips, seven did. A gap of a few tenths of a point between two models tells a buyer very little unless the benchmark also shows its uncertainty.
How to Test on Your Own Calls
Public benchmarks are still a reasonable way to build a shortlist. The decision should be made on audio that sounds like your customers:
- Build a test set from your own recorded calls, handled under your privacy and consent rules. Cover the accents, devices, line quality, and call types you see, and have people transcribe them carefully, because every model will be graded against that reference
- Score each model on that set and compare the result with its public score. Look at the gap as well as the number. A model whose accuracy barely moves between clean and real audio is more likely to hold up on calls it hasn't heard.
- Test enough audio that the differences on which you act are larger than the noise, and report a margin of error with each score
- Score the words on which your workflow depends separately: names, dates, account numbers and addresses. Zhong's column makes that case in detail.
- Time the final transcript. A voice agent usually waits for the final transcript before it replies, and in our benchmark the median wait between a caller finishing and the final text arriving ranged from 38 milliseconds to 652 milliseconds depending on the model.
- Run the test again whenever you change models or a vendor ships an update.
In our test, the model that topped the clean-clip ranking would have looked like the safe choice. On the phone conversations, its error rate was 65 percent higher than the leader's.
Tarush Agarwal is co-founder and CEO of Cekura.