.png)
.webp)
_logo%201.svg)
AI Summary
- Three transcribers of the same audio disagreed at 7.77–9.54% WER (Gref et al., 2022).
- Cleaning crowd transcripts from 10.91% to 4.94% error cut ASR error 9.52%.
- Manual speaker-boundary annotation runs at 15–30× real time.
- EU AI Act Article 10 regulates annotation; the Annex III deadline moved to 2 December 2027.
Audio annotation is the process of adding structured, machine-readable labels to recorded sound so that a model can be trained on it or measured against it. It turns unstructured audio files into structured, annotated data that machine learning models can learn from. Those labels include transcribed words, speaker turn boundaries, word-level timestamps, language and code-switching tags, sound event classes, and speaker attributes. Audio transcription is one of several audio annotation types. People use the two words interchangeably and lose the distinction that matters most.
That distinction decides which metric you can report. Word error rate needs a reference word sequence. Diarization error rate needs time-aligned speaker segments. Concatenated minimum-permutation word error rate needs both at once, and you cannot compute it on data collected for the first two. Speech models in 2026 cover more than 1,600 languages, while the human-verified evaluation data needed to check those claims covers a small fraction of that.
Introduction
In 2024, researchers at Cornell, Virginia Tech and NYU ran public-domain recordings through OpenAI's Whisper and counted how often it produced text with no counterpart in the audio at all. Their FAccT paper, Careless Whisper, reports the result. Roughly 1% of transcriptions contained fabrications, and 38% of those carried explicit harm: invented medical treatments, invented violence, invented racial descriptors. The hallucinations clustered around long pauses, so they fell hardest on speakers with aphasia and other speech impairments.
Speech engineering spent thirty years learning to handle a different failure mode. Older recognizers degraded visibly; garble announces itself. A fluent, grammatical sentence that was never spoken announces nothing, and catching one means comparing the output against a human-verified reference transcript of the same audio.
The audio AI market has solved coverage at a pace almost nobody predicted. A single open model now claims transcription across 1,600-plus languages, more than 500 of which had never been served by any recognizer. Verification has not kept up. Almost every accuracy claim in the category rests on a benchmark the claiming vendor built, and checking those claims independently requires reference data that is speaker-attributed, timestamped, multilingual and human-verified. You cannot download that. Someone has to produce it, at a cost you can measure, with an error rate you can measure, and since this summer with a statutory paper trail attached.
Audio annotation is the discipline that produces it. This guide covers what audio annotation outputs, the six types and the metric each one makes available, a working process that survives the cost figures, how to tell whether your labels are any good, why multilingual audio breaks pipelines that work fine in English, and what regulators now require on a timeline that shifted in July 2026.
What Is Audio Annotation, and What Does It Actually Produce?
Audio annotation produces a structured description of a recording that a machine can read: what was said, by whom, when, in what language, over what background noise, and with what vocal characteristics. It turns a folder of audio recordings into a training corpus, which is how speech recognition systems, natural language processing pipelines and voice assistants learn from human speech in the first place. Different projects need different subsets of that description. The subset you pick is a commitment: it fixes which evaluation metric you can compute later.
Most pages on this topic treat audio annotation and transcription as interchangeable, and that conflation is where audio data projects go wrong. A verbatim transcript supports word error rate and nothing else. The moment your product needs to know which of four people in a meeting said a sentence, or to place it at 04:12.380 instead of "somewhere in the third minute," the transcript is insufficient ground truth. No post-processing recovers what was never labeled.
Audio labeling and audio annotation mean the same thing; nothing turns on the choice of word. Scope is where the real decision sits. Labeling audio data for one purpose produces a dataset that supports one metric. High-quality audio datasets that serve several downstream models get built by someone deciding upfront to collect several label layers over the same recordings.
Here is the landscape, arranged by what each type demands of the annotator.
If you read down the third column, you'll find each row demands strictly more of the annotator than the row above it. Transcription needs words. Diarization needs time. Speaker-attributed transcription needs words, speakers and time simultaneously, which explains both its cost and why joint transcribe-and-diarize models changed what buyers ask annotation teams for. The metric you intend to report is a decision about data collection, and it has to be made before the first file is assigned.
Single-purpose tools tend to fail on the second project for the same reason. Real corpora get annotated for several things, by different teams at different times, over the same asset. Keeping those layers aligned to one timeline, one speaker registry and one guideline is a platform problem well before anyone starts comparing annotation interfaces.
What Are the Main Types of Audio Annotation?
These six types cover almost all production work, and each fails in a way the others do not.
Transcription
Audio transcription converts spoken words to written text as a reference word sequence, scored with word error rate or, for languages without clean word boundaries, character error rate. Speech-to-text transcription of clean read audio is the easy case. Conversational speech with overlap, background noise and several different speakers is where transcription tasks consume budget. The output is text data, and once it exists the whole natural language processing toolchain applies to it, which is part of why transcription gets treated as the default and the other layers get skipped.
The apparent simplicity is misleading. Every project has to decide how to handle filled pauses, false starts, repetitions, numbers, casing and punctuation, and those conventions move the measured error rate on identical audio. The MUCS 2021 multilingual and code-switching challenge at Interspeech showed the effect directly: transliterating a Hindi–English test set into a consistent script produced 22.72% WER against 24.66% for the same system on the same audio scored conventionally. Two points of error came from a transcription convention.
Speaker diarization
Diarization, or speaker diarisation in British usage, answers "who spoke when" without naming anyone, producing time-stamped segments with anonymous speaker identifiers. It is the speech labeling step that lets downstream analysis separate different speakers in a meeting or a support call. It is scored with diarization error rate and Jaccard error rate, and degrades sharply with speaker count, overlap and distant microphones. Our speaker diarization models guide covers the model landscape, DER measurement and failure modes in depth. Diarization is a distinct annotation product with its own ground truth requirement. Teams that treat it as a free by-product of transcription have made the most common scoping error in audio projects.
Speaker-attributed transcription
Joint speaker-attributed transcription asks for words, speaker identity and timing at once, and the metrics reflect that stacking. As von Neumann and colleagues set out in their review of meeting recognition evaluation metrics in IEEE/ACM TASLP, concatenated minimum-permutation WER adds speaker labels to the requirement and its time-constrained variant adds word-level timings on top. One system on one recording scored 38.2% cpWER and 51.0% tcpWER, a thirteen-point spread produced by asking the reference data for more.
How hard this gets depends almost entirely on recording conditions. The CHiME-8 DASR challenge reported tcpWER between 23.1% and 99.1% across scenarios, with diarization error from 10.3% on telephone-style speech to 60.0% on dinner-party recordings, and overlapping speech occupying 13.9% to 31.7% of talk time. A system that looks competent on clean two-party audio can be unusable on four people in a kitchen. Only condition-stratified reference data will tell you which one you have.
Sound event labeling
The time axis separates two tasks that often get conflated. Audio classification assigns a single label to an entire clip: this recording is a car horn, this one is a violin. Music classification by genre or instrument works the same way. Sound event detection marks specific sounds inside the clip with start and end times, so one recording can carry a dozen overlapping events. A label per file is cheap. Time-aligned labeling costs considerably more.
Both are scored with error rate and F1, and label quality varies enormously between public datasets. The FSD50K team behind the open dataset of human-labeled sound events required two independent raters to agree before accepting a label and estimated 94.3% of results correct; they also documented AudioSet exceeding 50% label error in roughly 18% of its classes. Two datasets built for the same task, with reliability separated by their annotation process.
Write one asymmetry into your guidelines. A study of label noise in sound event detection from the University of Western Australia found that raising the deletion rate degraded F1 steeply, while raising the insertion rate produced no significant degradation. Missing labels do real damage. Spurious ones mostly wash out. So instruct annotators toward recall and let adjudication strip the false positives. The paper is a preprint, though the direction matches the FSD50K experience.
Language and code-switching tags
Multilingual and code-switched audio needs span-level language identification and a documented script convention before anyone starts transcribing. Bilingual speakers switch mid-sentence and sometimes mid-word, and the guideline has to say what happens at the boundary. Without one, two annotators produce two different transcripts of the same utterance and neither is wrong.
Paralinguistic and emotion labels
Emotion, sentiment, intent and speaker-trait labels are the most subjective category, which promotes rater agreement from a secondary check to the primary quality signal. These are the metadata tags that make sentiment analysis on recorded speech possible: emotional tone, apparent intent, urgency, and the speech patterns that distinguish a frustrated caller from a merely fast one. They are also the only category with a hard legal boundary in the EU: Article 5(1)(f) of the AI Act prohibits inferring emotions from biometric data in workplaces and educational institutions, applicable since 2 February 2025. Establish where the model will be deployed before scoping the project.
How Do You Annotate Audio? A Working Process
Annotating audio data at volume is an operations problem that happens to involve software. The numbers dictate the process, so start there.
The organizers of the Third DIHARD Diarization Challenge needed speaker boundaries accurate to within 10 milliseconds, with turns split on pauses longer than 200 milliseconds. They measured what that costs a trained annotator working manually: "real time rates typically greater than 15X and sometimes exceeding 30X." They then abandoned fully manual boundary annotation for turn-level transcription plus forced alignment. A research challenge redesigned its own methodology because the labeling was too expensive.
Transcription costs land in a similar range. Where published figures differ, the task definitions differ, not the findings. The Slovenian ARTUR speech corpus, documented in Language Resources and Evaluation, took roughly 25 person-hours per audio hour at €300 per hour inside a €600,000 project covering 1,000 recorded hours. That corpus was built for linguistic research as much as for model training, which is worth noting because research-grade corpora publish their methodology and commercial ones rarely do, making them the only honest basis for this kind of data analysis. A 2025 preprint on building speech data for Bambara, a predominantly oral language, reported around 30 hours per audio hour in lab conditions and 36 in the field, and those were correction costs, not from-scratch transcription. At the other end, Amazon's LibriCrowd collected read English through a crowd platform at $6 per speech hour.
Do not average those figures. The tasks differ along four axes: read versus conversational speech, high-literacy versus predominantly oral languages, correction versus from-scratch work, and research precision versus commercial tolerance. Decisions made in a project's first week put it somewhere in that range and mostly keep it there.
Standardize, then segment
Standardizing audio data before annotation begins is the cheapest efficiency gain available. Converge the file formats, sample rates and channel counts on one specification. A queue mixing 8 kHz mono call recordings with 48 kHz stereo interview files makes every downstream comparison suspect, and it forces annotators to recalibrate their ears between assets.
Then handle audio segmentation: split long recordings into workable spans and preserve the mapping back to the original timeline. Segment boundaries look like a preprocessing detail and behave like an annotation decision. MUCS 2021 flagged misalignments between transcriptions and segment start and end times as a recurring data problem, and every downstream timestamp inherits whatever the segmentation got wrong.
Write the guidelines before the first assignment
Conventions decide the metric, as the transliteration result above shows, so they belong in a written specification covering disfluencies, overlap, inaudible passages, numbers, casing, punctuation, code-switch boundaries and non-speech events. Add a rule that silence gets its own explicit label. Long pauses are where generative models fabricate text, and blank silence gives the review pass nothing to check against.
The sampling plan belongs in the same document. Data diversity decides whether the resulting model works outside the room it was recorded in: a recognition model generalizes to the extent that its training audio spanned accents, recording environments, microphone types and noise conditions. Decide that composition deliberately and write it down. A corpus assembled from whatever audio files were easiest to obtain will encode that convenience as a blind spot, and nobody discovers it until deployment.
Run a focused pilot before you scale
Test that specification on a small, deliberately awkward sample before committing the full corpus. A pilot of a few hours costs almost nothing against a 1,000-hour project, and it surfaces the disagreements an abstract specification never anticipates: how to mark a speaker who trails off, what counts as one event versus two, which of two plausible spellings wins. Catch those ambiguities in the pilot and you stop paying multiple annotators to resolve them inconsistently for the next six months.
Capture metadata alongside the labels
Labels describe the speech. Metadata describes everything around it: recording device, channel, environment, speaker demographics where consent allows, language variety, emotion or intent tags, and the provenance of the file itself. Rich metadata gives a model context awareness that transcripts alone cannot supply. It also makes stratified evaluation possible later, and you cannot report accuracy by accent, by microphone or by noise condition unless somebody recorded which was which at annotation time.
Pre-label with a model, then review
At 15–30× real time, manual annotation from scratch is defensible only for small reference sets. Everything at enterprise scale runs model output through human correction, and AI-assisted annotation is the only structure that survives these numbers. Judge scalable solutions in this category on how well they support the correction pass; the quality of the first-pass model matters less than people expect.
Two caveats attach. Correction is not free, and the Bambara figures above were correction costs. It also introduces anchoring, where reviewers wave through fluent-but-wrong output. Whisper's hallucinations are fluent by design, so reviewers have to listen to the audio and not just proofread the transcript.
Review, adjudicate, and measure
A single reviewer pass gives you a second opinion and no tiebreaker. Amazon's transcript-cleaning work used two distinct mechanisms in sequence, and the second recovered more error than the first. Consensus validation is the working pattern: assign a defined share of assets to multiple annotators, treat agreement as the acceptance signal, and send disagreements to an adjudicator with authority to set the final label.
A mature annotation process also runs three things continuously. Written guidelines get versioned whenever they change. Audits run on a scheduled random sample, not only on the assets someone thought to flag. And quality checks stay tight enough to catch drift within days instead of at delivery. Record who decided what, because that record doubles as your Article 10 documentation.
How Do You Know Your Audio Labels Are Any Good?
Researchers building ASR for German oral-history archives had three independent transcribers correct the same ten hours of interview audio, then measured the transcribers against each other. Their multi-staged cross-lingual acoustic model adaptation paper reports pairwise disagreement between 7.77% and 9.54% WER, concluding: "we estimate a human word error rate of 8.7 %." The best adapted system in the same study scored 15.6% WER on clean audio and 23.9% on noisy.
Human transcription carries its own error bar, and on difficult audio that bar is wide enough to swallow the differences vendor benchmarks are built to advertise. The paper is a preprint and the figure is specific to German conversational archive material, so treat 8.7% as an order of magnitude and not a constant. The operational point survives either way. If you have not measured inter-annotator agreement on your own audio, you do not know your reference set's error rate, and every model comparison you run on it inherits that unknown.
Annotation accuracy propagates directly into model performance, and one study puts a number on what high-quality annotation is worth. Amazon researchers built LibriCrowd from 100 hours transcribed by 4,433 crowd workers, measured a 10.91% transcription word error rate, then cleaned it twice. Confidence-based relabeling cut error 22%, to 8.48%; word-level voting cut it a further 42%, to 4.94%. Their study of the impact of transcription errors on ASR measured the downstream effect at a 9.52% relative reduction in model word error rate. Scaling model size did not close the gap. Cleaner labels bought accuracy that a larger model could not, which is the number to bring to the next architecture meeting.
Three design choices separate an annotation operation from a transcription queue.
Require agreement rather than assume it
FSD50K's protocol is a reusable template: two independent raters had to agree before a label was accepted, and 3,390 of 59,981 candidate labels failed that bar across a pool of more than 350 annotators. Publish agreement coefficients per class, per language and per annotator. When a coefficient falls, suspect the guideline before you suspect the annotators.
One honest limit. Agreement coefficients exist for transcription and sound event labeling. For speaker-boundary annotation specifically, no published figure was available for this guide. DIHARD III's retreat from manual boundary annotation demonstrates cost, and cost is not disagreement.
Isolate the data variable
NIST's 2024 Speaker Recognition Evaluation separates fixed and open training conditions so that systems can be compared without training-data differences confounding the result. Borrow the structure. Hold the evaluation set constant, change one thing at a time, and a shift in accuracy becomes attributable instead of mysterious. NIST's test, evaluation, validation and verification programme is candid that measurement science for AI remains unfinished, documenting "strengths and limitations of metrics in use or being proposed."
Keep a private evaluation set
The Stanford AI Index 2026 technical chapter records that evaluations designed to be challenging for years now saturate in months, and that benchmark audits routinely find inadequate documentation and no replication scripts. Public benchmarks tell you a model is competitive in general. Whether it works on your calls, your microphones, your accents and your vocabulary is a question only a private, human-verified set built from your own conditions can answer.
One related note on published accuracy claims: leading speech vendors publish benchmarks in which their own model wins, and two of the most-cited disagree with each other in each publisher's favour. When you cite an ASR or diarization accuracy comparison, name the party that ran it, every time.
Why Do Multilingual and Accented Audio Break Annotation Pipelines?
Coverage and verification have come apart, and the gap is widest in the languages that most need the technology.
Meta's Omnilingual ASR release extends transcription to more than 1,600 languages, "including over 500 never before served by ASR," against Whisper large-v3's 99. Roughly a sixteenfold expansion, and a real achievement. Most of it is also unverifiable. A language with no prior ASR support has no held-out evaluation corpus and no human-verified reference transcripts, which leaves nothing to score the new coverage against.
The release's own quality audit is sharper than any outside critique. Cross-vendor validation of 206 languages found 20 carrying misattributed language codes. Among transcript defects flagged as critical, audio-and-transcript mismatch accounted for 51.18%, incomplete or summarized text 21.97%, and wrong writing system 10.51%. The team that published the coverage claim documented it failing their own QA pass, on the fraction of languages they could check at all.
Benchmarks confirm what thin data does to accuracy. ML-SUPERB 2.0, presented at Interspeech 2025 across 149 languages and 93 varieties, reported character error rates of 76.2 for Whisper and 69.3 for OWSM on unseen languages, with dialectal material roughly doubling error against standard varieties (XEUS moved from 40.2 to 68.7 CER). The authors flag an annotation problem underneath the numbers, noting that "some data sources do not clearly indicate which written standard they represent." When reference transcripts silently mix orthographic standards, the measured error rate is partly a measurement of the annotation.
The Interspeech 2025 MLC-SLM Challenge put 78 teams against roughly 1,604 hours across 11 languages and five English accents, and put a price on human-labeled structure. The winning system scored 16.53% tcpMER with its own diarization against 9.60% MER when handed oracle speaker labels. Nearly seven points of error sat in the segmentation alone.
Accent gaps in widely-spoken languages are the best-documented version of this problem. A Stanford-led team tested five commercial systems from Amazon, IBM, Google, Microsoft and Apple on more than 2,000 samples of recorded interview speech and found, as Stanford News reported, that "systems misunderstood 35 percent of the words spoken by blacks but only 19 percent" spoken by white speakers. Effectively unusable output, WER above 0.5, occurred in more than 20% of samples from Black speakers against under 2% from white speakers. The peer-reviewed PNAS paper traced the disparity to acoustic modeling, not language modeling, which places its origin in the training audio itself. Dialect-stratified annotation is the corrective, and it is an engineering fix rather than a gesture.
Open crowd-sourced corpora narrow the gap unevenly. MIT Technology Review's account of Mozilla Common Voice puts the project at roughly 31,000 hours across about 180 languages from more than 900,000 volunteers, with English at 3,554 hours from 94,665 speakers and Finnish at 22 hours from 231. Contributors skew young and male. Volume alone does not produce representativeness.
Curation does. IEEE Spectrum's report on te reo Māori voice models describes a model trained on 7 hours 45 minutes of audio verified by 68 fluent speakers, reaching 6.78% word error rate. Under eight hours of verified, community-governed audio outperformed what far larger scraped corpora deliver for comparable languages, and it was licensed under community guardianship rather than released openly.
So an annotation operation running multiple languages needs per-language reviewer pools instead of one global queue. It needs acceptance thresholds set per language and dialect, because a project-wide average hides the worst performer. And it needs stratified sampling that puts every accent group in the evaluation set at a size supporting a meaningful error bar. The same logic applies inside a single language, where regional speech patterns move error rates by double digits.
What Do Regulators Require of Audio Data?
Annotation entered EU law as a named activity. Article 10 of the AI Act, on data and data governance, lists "annotation, labelling, cleaning, updating, enrichment and aggregation" among the data-preparation operations that providers of high-risk systems must document. The same article requires training, validation and test sets to be relevant, sufficiently representative and, to the best extent possible, free of errors, with explicit account taken of the geographical, contextual and behavioural setting of use. Article 10(5) permits processing special-category data solely to detect and correct bias, subject to pseudonymisation and deletion once corrected.
Read that alongside the 35%-versus-19% finding and the consequence is concrete. Accent and dialect representativeness in a speech corpus is now a documented statutory obligation for high-risk systems, and the bias-detection carve-out exists so that teams can collect the demographic information needed to prove it.
Voice and speech are not named in Annex III. Speaker identification and voice-based emotion recognition fall under the high-risk regime through the Article 3 definition of biometric data, which is an interpretation rather than an explicit listing, so word it that way in your own documentation.
Check the timeline before you plan against it. It moved this summer: the AI Omnibus amendment entered into force on 27 July 2026 and pushed the high-risk deadlines back, and the Commission's own regulatory framework page now carries the dates below. Anything published before August 2026 that tells you Annex III obligations landed on 2 August 2026 is describing the original calendar.
Sixteen months of extra runway is not a reason to defer the work. Building a representative, dialect-stratified, documented audio corpus takes longer than that if you are starting from an undocumented one, and the Article 50 marking duty is already live.
The UK ICO's biometric data guidance draws a line most vendor pages blur: a voice recording is not automatically biometric data. It becomes biometric data when processed specifically to identify someone, and only then does special-category status attach. Transcribing a call and building a voiceprint from it are separate processing activities carrying separate obligations.
Synthetic audio now needs a label of its own. Speech synthesis has reached the point where a cloned voice passes casual listening, and the FTC's Office of Technology, reviewing approaches to AI-enabled voice cloning, concluded there is "no silver bullet to prevent the harms posed by voice cloning." With Article 50 marking obligations live, provenance metadata distinguishing recorded from generated audio belongs in the annotation schema itself, not in a spreadsheet somebody maintains alongside it.
Where Is Audio Annotation Actually Used?
Adoption is broad; the constraint is data. The Stanford AI Index 2026 economy chapter records 88% of surveyed organizations using AI in at least one function, up from 78%, with text-to-speech appearing in 1,900 US AI job postings in 2025. Deloitte's survey of 2,770 senior leaders across 14 countries found data-related issues causing 55% of organizations to avoid certain generative AI use cases altogether. Deloitte sells AI implementation services, so take the figure and leave the recommendations.
Spend follows. Menlo Ventures put enterprise generative AI at $37 billion in 2025 against $11.5 billion the year before. Menlo is a venture firm with portfolio exposure, so trust the growth direction more than the precise multiple. Its healthcare ambient-scribe figure of $600 million is the closest defensible breakout for a voice-specific category that anyone publishes. On the supply side, a16z counts 90 voice agent companies through Y Combinator since 2020, with voice-building companies at 22% of the most recent class; the same caveat about investor incentives applies.
Call centers and conversational AI agents need speaker-attributed transcripts across accents and multiple languages, plus intent and sentiment labels, and evaluation sets built from real call audio instead of studio recordings. Virtual assistants and smart speakers sit in the same bucket. A voice command is only interpretable if the underlying model saw annotated data covering how people actually phrase requests, mid-sentence corrections and background noise included. Healthcare and clinical documentation is where the hallucination problem bites hardest: a Whisper-based clinical tool used by more than 30,000 clinicians across 40 health systems has transcribed an estimated seven million medical visits, and one research team found 187 hallucinations in over 13,000 clear audio snippets. Media, broadcast and accessibility work needs verbatim transcripts with timing precise enough for caption placement. The same output does double duty in educational accessibility and language learning, where a time-aligned transcript of recorded speech becomes study material and not just a search index. Video data pulls in the same direction: captioning, dubbing and multimodal search all need the audio track annotated to the same standard as the picture, so teams working with audio and video assets end up running both pipelines against one timeline. Security and voice biometrics needs labeled genuine-versus-spoofed audio, now including adversarial cases: IEEE Spectrum's coverage of inaudible audio attacks reports standard defences reducing attack success by only 7%.
Subject matter varies; shape does not. Each of these involves several annotation types over the same recordings, several languages, and several teams whose output has to stay consistent for as long as the model is in production.
How Do You Choose an Audio Annotation Tool?
Derive the requirements from the findings above and the specification writes itself. Feature lists are the wrong starting point.
Start with what your metric demands. Waveform display and segment-level timing get you WER, DER and cpWER, which covers most speech work. Word-level timestamping is a harder requirement and you only need it if you intend to report tcpWER, so decide before you start, because no post-process recovers it. Spectrogram view matters for sound event work, where overlapping events show up visually and slip past the ear; on speech-only projects the waveform is the working surface. Overlapping-speaker support is non-negotiable if your audio is conversational, since overlap runs from 13.9% to 31.7% of talk time.
Then the ergonomics, which compound across thousands of hours: keyboard-driven navigation between segments, segment looping for the passages that need three listens, and zoom that survives a two-hour recording.
Then the operational half, which is where second-year projects live. Multiple annotation layers over one asset, since transcription, diarization, events and language tags accumulate on the same audio. Model-assisted pre-labeling with human correction, because 15–30× real time does not scale. Multi-pass review with explicit adjudication and a recorded decision. Inter-annotator agreement computed and reported per class, per language and per annotator. Reviewer routing that respects language, with language-specific acceptance thresholds. Metadata as first-class schema fields, not a free-text notes column. And a complete audit trail: who labeled what, who reviewed it, what changed, when, under which version of the guidelines. That last item has a date attached — 2 December 2027, for anything in scope of Annex III.
Kili added Audio as a modality in August 2026, currently in beta with access on request. The interface is purpose-built for the diarization-plus-transcription core: draw timed segments on the waveform, tag each with a speaker, transcribe each segment, and add asset-level jobs for things like language or recording quality. Model output imports as pre-labels so annotators correct a draft instead of starting from a blank asset, which is the only structure that survives 15–30× real time.
What makes it an operation rather than an audio tool is the layer around it: multi-step review workflows with named reviewers per stage and configurable send-back, typed asset metadata you can filter and assign on so a language field becomes a routing key, project and organization roles, and an API and Python SDK for the parts you automate. The same ontology, workflow and roles carry across image, video, text, PDF and geospatial projects, which is what matters when audio is one workstream in a programme rather than the whole programme.
Two limits worth knowing while Audio is in beta. Consensus scoring and honeypot are not yet available on audio projects, so agreement measurement on speech sits outside the platform for now. And segment-level timing supports WER, DER and cpWER but not word-level tcpWER. If either is load-bearing for you, ask us about the beta roadmap before you scope the project.
Running one annotation type in one language, you will not need most of this. Running several projects, data types and languages with distributed reviewers, these are the parts that decide whether the operation still works in its second year.
Whatever you choose, the demo is not the test. The test is whether the tool lets you answer, six months from now, why a specific label in your evaluation set says what it says.
Conclusion
Nothing in the current incentive structure closes the verification gap, so it will not close on its own. Coverage is announceable. A model that reaches 1,600 languages makes a headline; an evaluation corpus for the 500th of those languages makes none, and building one costs 25 to 36 human hours per audio hour of somebody's careful work. Coverage keeps expanding, verification keeps lagging, and vendor benchmarks that each conclude in the vendor's favour fill the distance between them.
For teams deploying artificial intelligence against real audio content, that gap is the whole problem, and it is an annotation problem. Amazon's result is the one to sit with: cleaning transcripts from 10.91% to 4.94% error bought accuracy that scaling the model did not. Label quality was the binding constraint, not a preprocessing detail. The te reo Māori result arrives at the same place from the opposite direction, where under eight hours of verified, community-governed audio beat what scraped corpora deliver at hundreds of times the volume.
A regulatory floor is being poured under this as well. From 2 December 2027, providers of Annex III high-risk systems in the EU have to document their annotation operations and show their data accounts for the setting of use. The July 2026 omnibus moved that back from 2 August 2026. Teams that already measure agreement, route by language and log adjudication decisions will find compliance describes what they already do. For everyone else the documentation cannot be reconstructed after the fact, since the decisions it describes were never recorded, and sixteen extra months buys less than it looks like.
The move for 2027 planning is a small one. Build a private, human-verified evaluation set on your own audio conditions, with agreement measured and dialects stratified, before you commit to a model. No competitor can buy that asset, no vendor can benchmark it on your behalf, and no larger model substitutes for it.
Resources
Annotation quality, cost and label noise
- Multi-staged cross-lingual acoustic model adaption for robust speech recognition in real-world applications – Gref, M., Matthiesen, N., Schmidt, C., Behnke, S., & Köhler, J. (2022). Source of the 8.7% human word error rate estimate. Preprint.
- Impact of transcription errors on automatic speech recognition – Gao, J., Sun, H., Cao, C., & Du, Z. (2023), Amazon. LibriCrowd and the 9.52% relative downstream gain. Preprint.
- Speech corpus ARTUR: Design and creation – Verdonik, D., et al. (2025). Language Resources and Evaluation, 59(3), 1899–1924. Per-hour transcription cost.
- Building speech data for Bambara – arXiv (2025). Lab and field correction costs for a predominantly oral language. Preprint.
- FSD50K: An open dataset of human-labeled sound events – Fonseca, E., Favory, X., Pons, J., Font, F., & Serra, X. (2022). IEEE/ACM TASLP, 30, 829–852. Two-rater agreement protocol.
- Investigating the effects of label noise on sound event detection – Zhang, Y., Huang, D., & Togneri, R. (2024). Missing versus spurious label asymmetry. Preprint.
Metrics, benchmarks and challenge data
- The Third DIHARD Diarization Challenge – Ryant, N., et al. (2021). Interspeech 2021. The 15–30× real-time annotation figure.
- Meeting recognition evaluation metrics – von Neumann, T., Boeddeker, C., Delcroix, M., & Haeb-Umbach, R. (2025). IEEE/ACM TASLP, 33, 3174–3188. cpWER, tcpWER and ORC-WER defined.
- The CHiME-8 DASR challenge – Cornell, S., Park, T., Huang, S., Boeddeker, C., et al. (2024). 8th CHiME Workshop. Scenario-dependent tcpWER and DER.
- NIST 2024 Speaker Recognition Evaluation (SRE24) – National Institute of Standards and Technology. Fixed versus open training conditions.
- AI test, evaluation, validation and verification (TEVV) – National Institute of Standards and Technology (2026). Measurement science for AI.
- AI Index Report 2026, Chapter 2: Technical performance – Stanford Institute for Human-Centered AI. Benchmark saturation and audit findings.
Multilingual, accented and low-resource speech
- Omnilingual ASR: Open-source multilingual speech recognition for 1600+ languages – Keren, G., et al. (2025), Meta AI. Coverage claims and the language-code validation audit. Preprint.
- ML-SUPERB 2.0 – Chen, W., et al. (2025). Interspeech 2025. Unseen-language and dialectal character error rates.
- Interspeech 2025 MLC-SLM Challenge – Mu, Y., et al. (2025). Oracle-label gap in multilingual conversational speech.
- MUCS 2021: Multilingual and code-switching ASR challenges for low resource Indian languages – Diwan, A., Vaideeswaran, R., Shah, S., et al. (2021). Interspeech 2021. Transliteration and measured WER.
- Racial disparities in automated speech recognition – Koenecke, A., Nam, A., Lake, E., et al. (2020). PNAS, 117(14), 7684–7689.
- Automated speech recognition less accurate for Black speakers – Stanford News (2020). Accessible summary of the PNAS study.
- How this grassroots effort could make AI voices more diverse – Heikkilä, M. (2024). MIT Technology Review. Common Voice language imbalance.
- Indigenous AI voice models and Māori data sovereignty – Winkless, L. (2026). IEEE Spectrum. Quality over volume, and community guardianship.
Hallucination, safety and adversarial audio
- Careless Whisper: Speech-to-text hallucination harms – Koenecke, A., Choi, A. S. G., Mei, K. X., Schellmann, H., & Sloane, M. (2024). ACM FAccT 2024.
- Researchers say an AI-powered transcription tool used in hospitals invents things no one ever said – Scripps News (2024). Clinical deployment scale.
- Sounds unheard by human ears can hijack AI models – Gent, E. (2026). IEEE Spectrum. Inaudible adversarial audio.
- Approaches to address AI-enabled voice cloning – Federal Trade Commission, Office of Technology (2024).
Regulation and data protection
- EU AI Act, Article 10: Data and data governance – Regulation (EU) 2024/1689. Annotation and labelling as regulated operations.
- EU AI Act, Article 5: Prohibited AI practices – Regulation (EU) 2024/1689. Emotion inference in workplaces and education.
- EU AI Act, Article 50: Transparency obligations – Regulation (EU) 2024/1689. Machine-readable marking of synthetic output.
- Biometric data guidance: Key data protection concepts – Information Commissioner's Office (2024). When a voice recording becomes biometric data.
- AI Act regulatory framework – European Commission, Directorate-General for Communications Networks, Content and Technology. Current application dates as amended by the AI Omnibus, in force 27 July 2026.
Market and adoption context
- AI Index Report 2026, Chapter 4: Economy – Stanford Institute for Human-Centered AI. Organizational adoption and speech-related job postings.
- State of Generative AI in the Enterprise, Q3 – Deloitte (2024). Data issues blocking generative AI use cases.
- 2025: The state of generative AI in the enterprise – Tully, T., Redfern, J., Das, D., & Xiao, D. (2025). Menlo Ventures.
- AI voice agents: 2025 update – Moore, O. (2025). Andreessen Horowitz.
Related Kili guides
- Speaker diarization models guide: Benchmarks and failure modes – Kili Technology (2026).
- Our complete guide to video annotation – Kili Technology (2026).
Frequently Asked Questions
What is audio annotation?
Audio annotation is the process of adding structured, machine-readable labels to recorded sound so it can train or evaluate a model. Labels include transcribed words, speaker turn boundaries, timestamps, language tags, sound event classes and speaker attributes. It produces the ground truth that speech model accuracy is measured against.
What is the difference between audio annotation and transcription?
Transcription is one type of audio annotation. It produces a verbatim word sequence and supports word error rate. Audio annotation also covers speaker boundaries, word-level timestamps, language and code-switch tags, sound events and paralinguistic labels, each of which supports a metric that a transcript alone cannot compute.
How long does it take to annotate one hour of audio?
Published figures range from about 25 person-hours per audio hour for the Slovenian ARTUR corpus to 30–36 hours for Bambara field recordings, with crowd-collected read English at roughly $6 per speech hour. Manual speaker-boundary annotation to research precision runs 15–30 times real time.
What are the main types of audio annotation?
Six types cover most work: audio transcription, speaker diarization, speaker-attributed transcription, audio classification and sound event detection, language and code-switch tagging, and paralinguistic labels such as emotion, intent and sentiment. Each requires different ground truth and supports a different accuracy metric.
What is the difference between audio classification and sound event detection?
Audio classification assigns one label to a whole clip, which is how music classification by genre or instrument works. Sound event detection marks specific sounds inside the clip with start and end times, so several events can overlap in one recording. Detection needs time-aligned labeling, which costs more to produce.
What is the difference between WER and DER?
Word error rate measures transcription accuracy against a reference word sequence. Diarization error rate measures how much of the audio is assigned to the wrong speaker or missed entirely, against time-aligned speaker segments. They need different ground truth, so the metric you plan to report determines what you must annotate.
Is audio annotation data considered biometric data?
Not automatically. Under UK ICO guidance, a voice recording becomes biometric data when it is processed specifically to identify an individual. Transcribing a call is ordinary personal data processing; building a voiceprint to recognise the speaker triggers special-category status and its additional lawful-basis requirement.
What makes multilingual audio annotation harder?
Thin reference data, inconsistent orthographic standards and dialect variation. ML-SUPERB 2.0 measured character error rates roughly doubling from standard to dialectal varieties, and Meta's Omnilingual validation found 20 of 206 checked languages carrying misattributed language codes. Per-language reviewer pools and language-specific acceptance thresholds are the working response.
Ready to Build Audio Ground Truth You Can Defend?
Kili now supports audio annotation in beta: timed segments on a waveform, speaker tagging, segment transcription and asset-level classification, with model pre-labels imported for correction and multi-step review workflows around them, inside the same platform your image, video, text, PDF and geospatial projects already run in. Talk to our team about beta access and about your audio data operation.
![Best Computer Vision Annotation Tools for On-Premise Labeling [2026] Guide](https://cdn.prod.website-files.com/68da32b2041c593b0511a582/6a8ffe034889a4ba57aa64ba_Competitor%20Article%20-%20Listicle%203.webp)
![ASR Models Guide: Word Error Rate, Benchmarks and Failure Modes [2026]](https://cdn.prod.website-files.com/68da32b2041c593b0511a582/6a7d8d3743d1e7bbda9ae54e_ASR%20Models.webp)