← sources

Aggregate WER is a useless metric for voice agents in production

source: reddit (reddit:AI_Agents)published 2026-09-21 11:55ingested 2026-09-22 02:35

retained text

Our ASR was showing 95 percent accuracy so nobody questioned it but then someone actually checked the calls where customers read out IFSC codes and about half were wrong by at least one character. One wrong word in a twenty word sentence still makes sense, one wrong character in a bank code and it's garbage.



Indian speakers on 8kHz phone audio, M and N sound the same, S and F, B and D and P. Digits were worse bc double five and fifty five and five five mean the same thing and the model treated them as three different inputs. Surnames outside common ones were basically random, policy numbers had the same issue once letters and digits mixed and half our callers switch between Hindi and English mid sentence so the model would just commit to one language and butcher the rest, which is exactly when they're reading out the important stuff.



The problem was we'd been reporting one WER number for months to everyone including the collections team and it was hiding everything that mattered to them. A failed IFSC means a failed payment, a wrong name means a compliance flag, and the aggregate number told us none of that. Now we track per field separately and yeah the numbers are bad but at least when collections says the bot got the account number wrong again we can actually see it in the data.



We're looking at constrained decoding and a validation pass but no idea if that's patching over a model problem.

document fetched 2026-09-22 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1wmank9/aggregate_wer_is_a_useless_metric_for_voice