.png)
_logo%201.svg)
AI Summary
- Three commercial systems scored 16.5–19.2% WER on real contact-centre calls and 10.2–11.6% on Switchboard.
- Re-scoring six systems under different conventions moved one from 10.18% to 6.43%, with the audio and the model untouched.
- WER can exceed 100%, and a 10% figure has never meant 90% of words correct.
- Two careful LDC transcribers disagree with each other at 4.1–4.5% on conversational telephone speech.
- About 1% of Whisper transcriptions in one study contained invented sentences; 38% of those carried explicit harm.
Introduction
In 2020, researchers at Avaya and three universities ran three commercial speech recognition systems over 50 real customer service calls. The systems scored between 16.5% and 19.2% word error rate. The same three systems, in the same study, on the Switchboard benchmark: 10.2% to 11.6%. Nobody has repeated the measurement with current checkpoints, and the call recordings could not be released for legal reasons.
Word error rate is simple arithmetic: count the words a system substituted, the words it inserted that nobody said, and the words it dropped, then divide by the number of words in the reference transcript. Modern automatic speech recognition is routinely quoted at around 5% WER, which sounds like a solved problem and is the figure most buyers carry into a vendor conversation.
That six-point gap has three sources, and model quality is not among them. The audio is different: benchmark corpora are read, clean, close-talk and pre-segmented, and recorded conversations are none of those. The scoring is negotiable: standard practice rewrites both the reference and the system output before a single error is counted, and the rewriting rules vary by who is doing the scoring. And the reference has an error bar of its own, because it was produced by people, and the organisation that produces most of them has measured how much they disagree.
ASR in 2026 is meaningfully better than it was in 2023. Conformer encoders paired with language model decoders hold the top English accuracy slots, inference throughput has improved by more than an order of magnitude, and the thirty-second decoding window that shaped the previous generation of speech to text systems is no longer universal. The audio these systems are scored on has not moved at all.
What follows covers what an ASR system emits, which checkpoints and licences are in production use, how the headline number is assembled, where it stops predicting anything, and what a review process can recover.
What Does an ASR System Actually Produce?
Automatic speech recognition (ASR) systems convert speech into written text. Three stages get you there, and a team debugging a bad transcript needs to know which one produced the error.
Audio capture records sound waves through a microphone, at a sample rate most models expect to be 16 kHz. Feature extraction turns that raw audio into a spectrogram, a time-frequency representation from which acoustic features are read; almost every current system uses log-mel filterbank features here. The model then predicts a word sequence from those features. Everything downstream, including the tidy punctuation in the final transcript, is applied to that prediction.
The output has four parts, produced by different mechanisms and measured by different metrics. The token stream is the words themselves, and it is the only one WER scores. Timestamps align those words to the audio, produced either by the decoder or by a separate forced-alignment pass. Punctuation and casing are sometimes generated by the model and sometimes restored afterwards by a second model. Speaker attribution, which turns a wall of text into an exchange between an agent and a customer, is not part of transcription at all.
Whisper returns text and timestamps and carries no notion of who is talking. A pipeline that appears to hand back a labelled dialogue is joining its output to a speaker diarization model, and that join has an error profile nobody scored.
The Open ASR Leaderboard, the main controlled public comparison of open speech recognition models, strips punctuation and casing from both reference and hypothesis before scoring, alongside standardising numbers and spelling and removing fillers. Two ASR models with identical published WER can therefore differ completely in whether their raw output is readable, and no leaderboard or vendor table will tell you which.
Transcription matters because recorded speech is unreviewable at volume without it. Live captions and voice assistants are the consumer-facing applications. In contact centers, transcripts turn thousands of hours of calls into something a quality team can sample, a compliance team can produce on request, and an evaluation set can be built from. Natural language processing over those transcripts then drives intent classification, complaint detection and, increasingly, the evaluation of voice agents. In each case the transcript is an input to a decision, which is why the errors the metric hides tend to be the errors that cost something.
Which ASR Models Are in Production Use in 2026, and What Changed to Get Them There?
From four trained components to one
The earliest systems recognised spoken digits from a single speaker under laboratory conditions. What followed, and dominated until the mid-2010s, was the hybrid stack: hidden Markov models handling temporal structure, an acoustic model mapping acoustic features to phonemes, a pronunciation lexicon, and a separate language model scoring which word sequence was plausible. Each component had its own training objective and its own tuning, and the setup complexity was high enough that speech recognition stayed a specialism rather than something a general engineering team picked up.
End-to-end models collapsed that stack into a single network trained to map audio to text directly, and they now dominate modern ASR for accuracy. Acoustic modelling still happens, inside the network rather than as a separately optimised stage, and the language modelling that used to sit in its own decoder is either learned implicitly or supplied by an actual language model bolted onto the encoder.
Three end-to-end formulations matter. Connectionist temporal classification (CTC) maps audio frames to text labels frame-synchronously and treats each output as conditionally independent, which makes it fast and slightly weaker. The recurrent neural network transducer (RNN-T) adds a prediction network conditioned on what has already been emitted; because it is frame-synchronous it handles real-time audio input, which is why real time transcription and on-device deployments are usually transducers. Encoder-decoder transformer architectures, the Whisper lineage, run an encoder model over the whole utterance and decode autoregressively, buying accuracy and broad multilingual coverage at the cost of offline operation. The newest family, speech-language models, embeds audio inputs directly into large language model architectures so that transcription becomes one capability of a general model.
The five families in production
The leaderboard's architecture census puts numbers on the distribution. Across the 60 open models in the revision that reported it, 24 use Conformer encoders, 20 use Whisper encoders and 13 use self-supervised encoders, while the decoders split 26 transformer, 20 CTC, 8 RNN-T or TDT, and 6 LLM.
Architecture choice sets latency, accuracy and hardware cost at the same time, and no single option optimises all three. A transducer tuned for speed runs comfortably on edge devices. The large models pairing a Conformer with an LLM decoder post the highest English accuracy and need a datacentre GPU to run inference at any useful batch size.
Four things that changed since 2023
The top of the English accuracy table changed architecture. Conformer encoders with LLM decoders now hold the leading slots; the best CTC model ranks 23rd on WER and the best self-supervised-encoder model ranks 52nd of 64.
Inference speed collapsed, and the trade-off became explicit. In one harness on one GPU, Whisper large v3 posts a real time factor of 145.5 while a 0.6B transducer posts 3,386 — roughly twenty-three times faster, at a lower average WER. Efficiency is now reported next to accuracy, which the leaderboard authors note was previously rare.
The thirty-second window stopped being universal. Whisper's fixed context forced buffered decoding for anything longer, and its authors documented the consequences plainly: an inaccurate transcription in one window degrades the windows that follow, and beam search with temperature scheduling is needed to stop the decoder looping on a repeated phrase. Newer models either accept arbitrary-length input or carry context windows measured in tens of minutes.
Training data scaled past the point where anyone can audit it. Kaldi's Gigaspeech XL model was trained on 10,000 hours of conversational English, a figure that was large when it was published. Whisper was trained on more than five million hours of audio. The extra data bought broader language coverage and better behaviour on noisy audio; it also means nobody can tell you what is in the training set of the model transcribing your calls. How that data is distributed across languages then decides which of them the model can actually transcribe, on a log scale the section on demographic gaps returns to.
The checkpoints, and the licence trap that runs backwards
Open source models dominate serious deployments, and the licence attached to each checkpoint is the part teams skip and lawyers find later.
The Open ASR Leaderboard now evaluates 86 models across 12 datasets, which makes it the closest thing the field has to a standard benchmark for open systems. Its own paper concedes in the same breath that a single metric is not enough.
Commercial APIs including Rev AI and Google Chirp appear in those results and are deliberately absent from the table. The leaderboard gives the reason itself: the highest-ranking closed-source system placed sixth, and real time factor cannot be computed fairly for hosted systems at all, because upload latency and GPU allocation are not yours to control.
Licences split three ways. MIT for Whisper and Phi-4-multimodal, Apache 2.0 for Voxtral, IBM Granite Speech and Qwen3-Omni, and CC-BY-4.0 with an attribution obligation for Parakeet, Canary-Qwen and Kyutai. For anything embedded or white-labelled, that attribution requirement is a compliance step someone has to own.
The inherited restriction sits somewhere teams rarely look: in the evaluation corpora rather than the weights. The leaderboard's dataset table lists CoVoST-2 under CC-BY-NC-4.0 and TED-LIUM v3 under CC-BY-NC-ND 3.0, both non-commercial and the latter also no-derivatives, with SPGISpeech under a bilateral user agreement. A commercially permissive model can be ranked on data a commercial buyer cannot legally use to re-run the comparison. Provenance carries its own asterisk: NVIDIA states that 110,000 of Parakeet's roughly 120,000 English training hours are pseudo-labelled data derived from YouTube.
Model size decides where inference can run at all: a 0.6B transducer fits on edge devices, a 24B speech-language model does not, and that constraint usually settles the shortlist before accuracy gets a vote. For multilingual ASR, published English averages say nothing: broad multilingual coverage in a model card usually means the model has seen multiple languages, which is a weaker claim than performing well on low resource languages. Fine tuning against your own data, or measuring on it directly, is the only way to find out.
One property matters more to a regulated buyer than any WER figure in the table. Every open-weight checkpoint above runs locally with no outbound connection at inference, so whether audio leaves your network is a decision you make rather than one the vendor makes for you. Hardware is the binding constraint.
Why Isn't a 10% Word Error Rate the Same as 90% Accuracy?
Computing WER is arithmetic: substitutions plus insertions plus deletions, over reference words, after an alignment that finds the cheapest edit path between the two strings. Three decisions sit in front of that arithmetic, and none of them is visible in the reported figure.
Insertions break the intuition
WER is not the percentage of words a system got right. Insertions have no counterpart in the reference, so the numerator can exceed the denominator and WER can pass 100% — which is exactly what happened to one CHiME-8 baseline at 99.1%, and what happens routinely when a decoder falls into a repetition loop.
Transcription accuracy and word error rate also come apart even when the number behaves. A system can post 10% WER with the errors scattered harmlessly across fillers, or with every named entity in the call wrong. Identical scores, completely different transcripts.
Normalization: what gets rewritten before scoring
Scoring does not compare raw strings. NIST's toolkit applies a mapping file to both sides first, and the LDC documentation for those files sets out what it permits: alternate renderings of contractions, backchannel mappings, hesitations flagged as optionally deletable. So a system is not charged for writing "gonna" where a transcriber wrote "going to." The sclite documentation also describes weighted-word scoring, in which different words carry different costs, which almost nobody uses. Whisper's authors built and released their own English normalizer for the same reason and said so in the paper: to avoid penalising differences that carry no semantic content.
None of that is cheating. All of it is invisible downstream. In 2022, six systems were re-scored eight ways on the same NIST 2000 Switchboard test set, with the audio and the systems fixed and only the conventions moving.
The CHiME challenge organisers make the same point from the other direction. For their 2024 edition they modified the Whisper normalizer for their own scoring, removing number normalization to stay closer to verbatim and making the function idempotent, on the grounds that applying a normalizer twice should not produce a different string than applying it once. Scoring should be a well-defined mapping. It has not always been one.
There is no such thing as the WER of a system. There is only its WER on a named corpus under a named normalizer, and a percentage quoted without both is a number with no referent.
Every error costs the same
A deleted filler and a misrecognised account number each count as one error.
That would be pedantry if error types were evenly distributed. They are not. A University of Washington team re-analysed the human and machine transcripts behind the 2017 parity claims and found that at statistically indistinguishable average WER, the error distributions diverge sharply. On CallHome, the machine was 65% less likely to miss conversational words such as fillers and backchannels, and 81% more likely to misrecognise them. Humans were 3.5 times more likely to omit isolated hesitations. The machine was 199% more likely to insert content words and 80% more likely to misrecognise them.
Humans drop words they judge unimportant. Models turn them into other words. The two behaviours score the same and fail completely differently, which is why the same authors concluded that conversational speech technology may need ASR to be better than human transcribers rather than equal to them.
The same flattening hides entity errors. Because named entities are a small fraction of tokens, a system can get every drug name, product name and account balance wrong while aggregate WER barely moves.
Whose Transcript Counts as the Reference?
The reference transcript is treated as ground truth. It is produced by people, and the organisation that produces many of them has measured how much they disagree.
The LDC applied WER arithmetic between two of its own careful transcribers across genres. On read broadcast news, careful transcription produced 1.3% disagreement, 81% of it punctuation. On conversational telephone speech it reached 4.1 to 4.5 percent, and 95% of those discrepancies were adjudicated as judgement calls rather than errors. On 2007 conference-room meeting data, 64% of dually transcribed segments contained some disagreement.
That is the same magnitude as the best claimed conversational WERs, which is what went wrong with the human parity claims of 2016 and 2017. Both were declared by measuring human transcribers on the same audio, two labs did it, and they got different answers because they used different protocols. Microsoft's figures came from a commercial bulk-transcription pipeline with a second error-correction pass: 5.9% on Switchboard, 11.3% on CallHome. IBM commissioned three professional transcribers and took the best score after a fourth transcriber's quality check: 5.1% and 6.8%. The CallHome human reference differs by a factor of 1.66 between two competent teams on identical audio. IBM also noted that 36 of the 40 Switchboard test speakers appear in the training data.
Test-set selection does the same work. A 2026 preprint benchmarking human listeners against ASR on Dutch child, older-adult and Flemish speech found models matched or beat native listeners on curated stimulus sets, then scored roughly 8.4 points higher in WER terms on the full test sets those stimuli came from. The comparison is real; the conclusion depends on which subset you evaluate.
Where Does a Benchmark Number Stop Predicting Your Audio?
Start with the spread inside a single run. NVIDIA's model card reports per-dataset figures for one checkpoint under one protocol, greedy decoding, no external language model: 1.69% on LibriSpeech test-clean, 9.74% on GigaSpeech, 11.15% on Earnings-22 and 11.16% on AMI. A 6.6× spread with the model and the scoring held constant. The AMI figure is the individual headset condition, close-talk, not far-field. Add background noise and it moves again: at 0 dB SNR the average goes to 11.88% and AMI to 25.43%.
The contact-centre figures in the introduction are the same effect measured on production audio instead of a corpus.
Every condition below is excluded by construction from the corpora those numbers come from. LibriSpeech is studio-recorded read audiobook speech, and it is the figure that gets quoted. Benchmark segments contain speech by definition, so silence, hold music and IVR prompts never appear. Most corpora assume oracle segmentation and one close-talk channel. They are pre-segmented into short utterances, so long-form drift has nowhere to show up. Speaker pools are demographically narrow and largely native monolingual. Entities are a small share of tokens, so aggregate WER barely moves when a system gets all of them wrong. And almost no public corpus of modern VoIP contact-centre audio exists at all.
Corpus provenance accounts for much of the variance in these tables. LibriSpeech is clean audio: read audiobooks, studio conditions, one speaker. AMI is meeting audio from headsets and microphone arrays. Switchboard, CallHome and Fisher, the telephone corpora everyone still benchmarks on, are between twenty-five and thirty-five years old and describe landline conditions that no longer exist. A LibriSpeech figure is not a forecast of your own traffic.
Mitigation for the entity problem exists. Contextual biasing with a hotword list has been reported to cut biased WER by up to 88% relative while leaving other words largely unaffected, and entity-weighted metrics exist precisely because aggregate WER cannot see the transcription errors that cost the most.
Hallucination is a different kind of error
Edit distance was designed to count wrong words. It was not designed to count words that correspond to no audio at all.
A peer-reviewed FAccT study evaluated Whisper on recorded speech from people with aphasia and a control group, isolating spans that appeared in the transcript and nowhere in the recording. Roughly 1% of transcriptions contained entire fabricated phrases or sentences, and thematic coding found that 38% of the invented text carried explicit harm: perpetuating violence, inventing associations, asserting false authority. In a 6,046-segment subset, segments from speakers with aphasia showed 1.8% hallucinations against 1.1% for controls, and longer non-vocal periods predicted hallucination. The authors call their own count an undercount, because they only analysed segments producing different outputs across runs.
WER charges a fabricated sentence the same as a handful of ordinary substitutions. The population least able to contest an invented transcript is the one it happens to most.
The demographic gap in American English has narrowed and has not closed
The published evidence here is all American English, and the speaker categories are the ones the studies themselves used. A 2020 PNAS study compared sociolinguistic interviews from the Corpus of Regional African American Language, recorded in Washington DC, Rochester and Princeville with speakers of African American Vernacular English to varying degrees, against interviews from the Voices of California corpus recorded in Sacramento and Humboldt. Five US cities, two corpora, one variety of English.
Across those recordings, five commercial systems averaged 0.35 WER for Black speakers against 0.19 for white speakers, over 42 white and 73 Black speakers and 19.8 hours matched on age and gender. The gap persisted on identical phrases spoken by both groups, which locates it in the acoustic model rather than in vocabulary. What the acoustic model is failing on is dialect and pronunciation, and the authors' own framing is about the obstacles African Americans encounter when using these tools.
Nobody has re-run that analysis on current checkpoints with the original method. The nearest update, a CHI 2025 study applying Whisper large-v2 to audio from a US tutoring programme, found 0.48 against 0.39 across 12,572 propensity-matched utterances, and still 0.58 against 0.52 when matched on identical n-grams. Perplexity was lower for Black tutors' speech, which rules out a lexical explanation. Fine tuning narrowed the gap from 37% to 26% without closing it. Different corpus, different model, different matching procedure: do not subtract one ratio from the other.
None of it transfers to English as spoken elsewhere either. The Edinburgh International Accents of English Corpus collected 40 hours of conversation across more than 40 self-reported accents, and the best model tested there, trained on 680,000 hours, averaged 19.7% WER against 2.7% on US English read speech, with the sharpest drops on Jamaican and Nigerian English.
Across languages the disparity is larger and its mechanism is documented. Whisper's own paper reports a squared correlation of 0.83 between the log of a language's word error rate and the log of how many training hours that language had, and estimates that WER halves for every sixteenfold increase in data. Coverage is a data-distribution outcome, not a property of the architecture. At the bottom of that distribution the output is not a slightly worse transcript but an unusable one: a 2026 benchmark of ten multilingual models on Pashto, a language with 60 to 80 million speakers, found zero-shot Whisper WER ranging from 90% to 297%, with one configuration reaching 461% through decoder looping, while the best zero-shot result from any model was 39.7%. A 2026 study of six Southern Bantu languages with more than 80 million speakers between them reports the same pattern of foundation models exceeding 100% WER before fine-tuning, and adds a further result: no single model won across all six, with one architecture ahead on the Nguni languages and another on Sotho-Tswana, a split the averaged figures concealed. Corpus choice compounds it. ML-SUPERB 2.0, covering 141 languages across 15 corpora, reports per-language character error rate standard deviations between roughly 10% and 22%, and puts Urdu at 21.8% CER on Common Voice against 56.9% on FLEURS. Same language, same benchmark, two corpora.
Accent and language gaps deserve their own guide and will get one. For this article, treat every figure in the section as the shape of the problem and not its size in your data.
Disordered speech is better documented and improving, on American English corpora again. Best published WER on one dysarthric benchmark fell from 66.7% in 2010 to 17.8% in 2023. The Interspeech 2025 Speech Accessibility Project challenge scored submissions on WER and on a semantic score, which tells you what the organisers make of WER on its own.
What Happens to a Transcription Error Downstream?
The transcript is rarely the deliverable. Whatever reads it next inherits its errors.
The CHI 2025 team traced that cascade. A downstream classifier's F1 fell from 0.66 to 0.51 for Black tutors when run on ASR output rather than human transcripts, against 0.64 to 0.54 for white tutors, and a genuine measured advantage in discourse quality disappeared entirely once the pipeline ran on machine transcripts. The disparity grew as it moved down the pipeline.
How much error a pipeline can absorb depends on what it does with the transcript. CHiME-8's downstream summarisation evaluation found only weak correlation with transcription quality: on one meeting corpus, systems above 50% tcpWER produced summaries roughly on par with systems around 11%. Summarisation is forgiving because it aggregates. Compliance evidence, dispute reconstruction and the evaluation of voice agents are unforgiving, because they turn on specific words, and specific words are what entity errors remove. Where attribution matters as well as wording, cpWER and its time-constrained relatives measure the two jointly, and the gap between cpWER and plain WER isolates the cost of getting the speaker wrong.
Regulation pushes the same way. MiFID II Article 16(7) and FCA SYSC 10A require firms to record relevant client conversations and retain them for years, and a recording kept as evidence is only useful if the transcript of it is accurate and attributed. Article 50 of the EU AI Act became applicable on 2 August 2026, with deployer transparency duties set out in the European Commission's own FAQ. Vendor material regularly blurs one boundary here. Article 5(1)(f) prohibits emotion recognition in the workplace outright, subject to a narrow medical and safety exception, and legal commentary reads that as covering call centres. Inferring affect from a call recording is a different regulatory object from transcribing one.
The pattern holds across sectors that record customer conversations at volume. In banking and insurance the transcript functions as compliance evidence and as the record in a dispute. In retail and luxury it is how service quality gets assessed across boutiques and concierge lines. In telecom, utilities and healthcare it feeds agent coaching and, increasingly, the training and evaluation of voice agents that speak to customers directly. In none of them is the cost of an error proportional to the WER delta.
What Actually Reduces the Errors You Care About?
A careful human reference transcript costs about 25 times real time to produce, or 50 times with a verification pass, according to the same LDC study that measured transcriber disagreement. Quick transcription runs at 5 times. Labeled data for speech is expensive because the task is hard, and it is hard for the reasons the disagreement figures describe.
Correcting machine output beats typing from scratch, conditionally. A study of 160 crowdsourced participants put the crossover at roughly 30% WER: below it, starting from ASR output helps; above it, it hurts. That threshold comes from captioning and lecture tasks rather than conversational telephone speech, so treat it as a finding rather than a law. Current models sit below it on most recorded business audio, which makes correction against the output of pre-trained models the sensible default.
Post-editing improves the quality of the final transcript and not only the speed, and in that study plain from-scratch typing was the only interface that failed to improve on the automatic baseline. The benefit is also individual: in a subtitling study, two of three subtitlers got faster and one did not, with WER predicting per-segment productivity gain at a Spearman correlation of −0.676.
That variation is the empirical reason a single reviewer's output cannot be treated as ground truth. If two careful transcribers disagree at 4.1 to 4.5 percent on conversational audio, one reviewer's pass carries an error bar the size of the thing being measured. Agreement has to be computed, which in practice means gold-standard items seeded into the queue, a share of the audio reviewed twice with agreement scored, and a record of who changed what.
Hallucination adds a harder requirement. A fabricated span is invisible to anyone who cannot hear the audio, because it reads perfectly. An AP investigation into a Whisper-based clinical documentation tool, used across dozens of health systems, found that the vendor deletes the original recording once the transcript exists. Review without the source audio is proofreading, and proofreading cannot catch invention.
The leaderboard authors arrive at the same place from the other end. Published numbers tell you which models to shortlist and nothing about which one works on your traffic. Your own evaluation, built on your own data with your own speakers and channel conditions, is the only measurement that transfers.
Kili Technology's audio interface is built to those requirements. Reviewers work against the waveform with an ASR pre-annotation already in place, correcting the words and the speaker attribution in the same pass, which is the condition the post-editing studies found human correction reliable under. Consensus scoring and gold-standard items turn reviewer agreement into a measured quantity, and every correction stays attributable. When the recordings are retained under a regulatory obligation and cannot be sent to a third-party service, the same review process runs on-premise. And because one customer relationship generates calls, emails and chat logs from the same account, holding all three to a single set of data quality metrics is cheaper than maintaining three standards for it.
Attribution errors and transcription errors are separate failures with separate metrics. A review pass that fixes the words and leaves the turn boundaries wrong yields a transcript that is accurate and still unusable for anything depending on who said what.
Conclusion
The scoring conventions in this field are a documented admission, written into the tooling by the people who built it, that the human reference has a resolution limit and that charging a system for rendering a contraction differently measures nothing. The forgiveness is honest. Quoting the resulting percentage without the corpus and the normalizer that produced it is not.
So the procurement conversation has to change shape. Ask for the corpus, the normalizer and how named entities were scored alongside any figure a vendor gives you. Then set all of it aside and measure on your own recordings, because the answer will not transfer — a six-point swing between benchmark and production audio is the documented case, and it was measured on systems that had already been shortlisted.
The field's open questions point the same direction. There is no public WER on modern VoIP contact-centre traffic. No replication of the 2020 demographic study on current checkpoints. No plotted degradation curve for long form audio. No evidence at all about whether architectures without an autoregressive text decoder hallucinate at rates comparable to Whisper's.
None of those gaps closes with a better checkpoint. Every one of them closes with reference data built on real audio and held to a standard that survives inspection, and that work sits with whoever owns the recordings.
Resources
Metric Construction and Scoring Conventions
- Toward Zero Oracle Word Error Rate on the Switchboard Benchmark (Faria, Janin, Riedhammer & Adkoli, Interspeech 2022) – six systems re-scored eight ways on fixed audio
- sclite (NIST SCTK documentation) – alignment procedure and weighted-word scoring
- LDC RT-02 readme and Global Mapping File documentation – what normalization permits before errors are counted
- Transcription Methods for Consistency, Volume and Efficiency (Glenn, Strassel et al., LREC 2010) – inter-transcriber disagreement and real-time cost by method
Human Parity and Error-Type Analysis
- Toward Human Parity in Conversational Speech Recognition (Xiong et al., IEEE/ACM TASLP 2017) – the 5.9% and 11.3% human figures
- English Conversational Telephone Speech Recognition by Humans and Machines (Saon et al., Interspeech 2017) – the 5.1% and 6.8% human figures under a different protocol
- Revisiting Parity of Human vs. Machine Conversational Speech Transcription (Mansfield et al., Interspeech 2021) – error-type divergence at equal WER
- Benchmarking Human and Automatic Speech Recognition of Diverse Speech (Huisman et al., 2026) – test-set selection changes the conclusion; preprint
Benchmarks and Evaluation
- Open ASR Leaderboard (Srivastav et al., 2025) – standardised normalization, real time factor, architecture census, dataset licences; preprint
- Recent Trends in Distant Conversational Speech Recognition: CHiME-7 and 8 DASR (Cornell et al., 2025) – far-field tcpWER results and downstream summarisation analysis; preprint
- WER We Are and WER We Think We Are (Szymański et al., Findings of EMNLP 2020) – real call-centre audio against benchmark audio
- The Interspeech 2025 Speech Accessibility Project Challenge – WER scored alongside a semantic metric; preprint
ASR Models, Licences and Provenance
- Robust Speech Recognition via Large-Scale Weak Supervision (Radford et al., 2023) – Whisper's 30-second window, training scale and normalizer rationale
- NVIDIA Parakeet TDT 0.6B v2 model card – per-dataset WER, SNR and telephony rows, CC-BY-4.0 licence, training-data provenance
- IBM Granite 3.3 announcement – arbitrary-length audio input, Apache 2.0
- Voxtral technical report (Liu et al., 2025) – 32K context, Apache 2.0
- Qwen3-Omni technical report (Xu et al., 2025) – speech-language model family, Apache 2.0
Failure Modes and Mitigation
- Careless Whisper: Speech-to-Text Hallucination Harms (Koenecke et al., FAccT 2024) – hallucination rate and harm coding
- WhisperX: Time-Accurate Speech Transcription of Long-Form Audio (Bain et al., Interspeech 2023) – VAD chunking for long form decoding
- Timestamp drift across ASR and audio-language systems (Chou et al., 2026) – decoded time axis drifting from the audio; preprint
- How to Recognize New Words: Context Biasing Methods and Speech LLMs (2026) – biased-WER reductions for rare words and entities; preprint
- Lightweight Prompt Biasing for Contextualized End-to-End ASR (Ren, Shi & Li, 2025) – entity word error rate as a working metric; Microsoft-authored, in-house data
Fairness and Accessibility
- Racial Disparities in Automated Speech Recognition (Koenecke et al., PNAS 2020) – 0.35 against 0.19 WER across five commercial systems
- Cascading Effects of Bias in Automatic Speech Recognition (Ezema et al., CHI 2025) – disparity on a current model and downstream classifier impact
- Community-Supported Shared Infrastructure in Support of Speech Accessibility (Hasegawa-Johnson et al., 2024) – dysarthric speech WER and fine-tuning gains
- The Edinburgh International Accents of English Corpus (Sanabria et al., ICASSP 2023) – 40+ English varieties, 19.7% average WER against 2.7% on US read speech
- ML-SUPERB 2.0 (Shi et al., Interspeech 2024) – 141 languages across 15 corpora; per-language CER spread and the Urdu corpus discrepancy
- Benchmarking Multilingual Speech Models on Pashto (Rahman, 2026) – zero-shot WER from 90% to 297%, decoder looping past 400%; preprint
- Tone-Conditioned Curriculum Learning for Low-Resource Bantu Speech Recognition (Mokgosi et al., 2026) – above-100% zero-shot WER on six Southern Bantu languages, and per-language architecture differences hidden by averages; preprint
Human Correction and Review Economics
- The Effects of Automatic Speech Recognition Quality on Human Transcription Latency (Gaur et al., W4A 2016) – the ~30% crossover
- Optimizing Computer-Assisted Transcription Quality with Iterative User Interfaces (Sperber et al., LREC 2016) – post-editing improves quality, not only speed
- SubER: A Metric for Automatic Evaluation of Subtitle Quality (Wilken et al., 2022) – individual variation in post-editing productivity
Regulation and Reporting
- MiFID II Article 16(7) and FCA SYSC 10A – call recording and retention requirements
- EU AI Act, Article 50 – transparency obligations applicable from 2 August 2026
- European Commission FAQ on Article 50 – deployer duties in the Commission's own words
- AP investigation into a Whisper-based clinical transcription tool – source recordings deleted after transcription
Related Reading from Kili
- Speaker Diarization Models Guide: Benchmarks and Failure Modes [2026] – the attribution side of the same audio stack
- Top Data Quality Metrics for Assessing Your Data Labeling Quality
Ready to Build Better ASR Models?
When you are correcting ASR output at volume, review design carries more of the outcome than model selection does. Talk to the Kili Technology team about running transcript and speaker correction against the source audio, on your own infrastructure.
![Speaker Diarization Models Guide: Benchmarks and Failure Modes [2026]](https://cdn.prod.website-files.com/68da32b2041c593b0511a582/6a7599f6a13b83298c74e35c_Speaker%20Diarization%20Models.webp)
