Fine-Tuning Audio Models Guide: What Annotation Quality Decides [2026]

Introduction

In 2026, Lukas Wagner, Mario Zusag and Bernhard Thallinger at nyra labs ran a simple test. They took two reference transcripts of the same audio — both containing 100% of the same content words — and scored them against each other. On TED-LIUM, a benchmark built from TED talk recordings, the transcripts disagreed at 3.7% word error rate (WER). On AMI, a benchmark built from multi-speaker meeting recordings, they disagreed at 12.1%. One transcript recorded the false starts and filler; the other recorded what the speaker meant. Nothing about the recording itself had changed.

Their decomposition of transcription policy goes further than the disagreement figure. Reference choice alone moves Whisper's TED-LIUM score from 5.7% to 7.3%. And roughly 60% of its reported AMI word error rate turns out to be style, not information the model failed to recover. On the training side, the effect is worse than a scoring artefact. A corpus annotated somewhere between 15% and 85% verbatim produces a model that emits disfluencies unpredictably — with no user-facing switch to control it.

The annotation convention, in other words, is a hyperparameter. Most teams never declare it, never version it, and only discover its value by inspecting outputs after the fine-tune has finished. The authors sell speech products and release their own Whisper variant, so they have a commercial interest in this conclusion — that transcription style needs explicit control. The measurements are unusually clean, though, and the direction is confirmed elsewhere in this guide.

What follows is the practical sequence for fine-tuning a speaker-attributed transcription stack:

• why you'd fine-tune a speech-to-text model at all
• what the fine-tuning process mechanically involves
• how much labelled audio the published curves actually support
• which component to adapt first
• what the guideline has to specify before the first assignment
• where human annotators reliably fail
• whether you can trust a model's pre-annotations
• how to evaluate without fooling yourself
• when fine-tuning moves your legal position

The through-line is the same argument Kili's audio annotation guide makes at a higher level: the reference data sets the ceiling on a fine-tuned model, and everything else is negotiation below that ceiling.

Why Fine-Tune a Speech-to-Text Model Instead of Using the Base Model?

Large pre-trained audio models — Whisper, wav2vec 2.0, the pyannote family — learn general speech patterns from a bigger, more generic dataset than any single team could assemble. That generality is exactly why base-model output disappoints on real recordings. Automatic speech recognition (ASR) trained on read and broadcast speech has no particular reason to know your industry-specific terms, your acronyms, or your product names. It has even less reason to know how three people talk over each other in a badly miked meeting room.

Fine-tuning continues training those pre-trained weights on a domain-specific dataset, and it buys things a bigger base model can't. Domain-specific jargon is the clearest case. An ASR model fine-tuned on clinical dictation learns the medical terminology that a general model transcribes phonetically or drops. The same holds for legal, financial and industrial vocabularies. Low-resource languages are the second case. Teams have fine-tuned Whisper for languages with almost no labelled audio, Dhivehi among them, by building on the multilingual knowledge already in the pre-trained model instead of starting from scratch. Domain adaptation on your own audio files also teaches the model your microphones, your channel and the speech patterns of your actual speakers — not a benchmark's.

The same fine-tuning process serves work beyond transcription, too: speech translation, sound-event tagging, speaker diarization, command recognition for voice assistants. Those downstream tasks may need distinct model architectures, and teams often replace or add task-specific heads rather than inheriting them.

What fine-tuning does not do is rescue bad data. Training on a noisy, inconsistently labelled corpus degrades transcription accuracy relative to the base model, and the degradation stays invisible until someone reads the output. That failure mode runs through the rest of this guide.

What Does the Whisper Fine-Tuning Process Actually Involve?

The mechanics take less engineering than the planning meetings around them suggest. The Hugging Face Whisper fine-tuning walkthrough is the reference implementation most teams start from. It runs end to end on a free-tier 16GB T4 GPU in under an hour, on roughly ten hours of labelled data. If a full fine-tuning run costs an hour of GPU time, the run is not the bottleneck in your project plan — producing the ten hours of consistently annotated audio is.

Choosing a base model and a pre-trained checkpoint

Whisper ships in several sizes, and the choice trades transcription accuracy against memory usage and latency. Whisper small is the usual starting point for a first fine-tune because it fits on one consumer GPU; whisper-medium and large need parameter-efficient methods or quantisation just to fit. The pre-trained checkpoint also constrains what you can adapt it to, through its language coverage, its sampling-rate assumption and its tokenizer — so pick the starting model deliberately, not by download count.

Getting the audio data ready

Whisper expects an audio signal sampled at 16kHz. So the first data-processing step is resampling every audio clip and confirming the sampling rate matches what the model expects. Get this wrong and you get a run that trains cleanly and transcribes nonsense.

From there, the Whisper feature extractor handles feature extraction. It pads or truncates each audio array to 30 seconds and converts it to a log-mel spectrogram — the input features the encoder consumes. The transcript goes through the tokenizer separately, and the resulting label IDs become the decoder's targets.

Preparation work that pays off before any of this runs:

• Normalise loudness across audio files, so the model isn't learning your recording levels instead of your speech.
• Segment long audio samples into consistent chunks, using silence-aware splitting rather than fixed cuts (see the boundary-error figures below).
• Apply audio augmentation, such as added noise, reverb or speed perturbation, where the target domain is noisier than the training set.
• Keep the labelling clean and the examples diverse. A dataset that reflects real-world variability in speakers, channels and spoken utterances beats high quality data on a narrow slice, which produces a model that only works on that slice.

The data collator and training arguments

Speech-to-text training needs different padding methods for the two halves of each example. That's why the walkthrough defines a custom data collator instead of using the default one. Input features are already fixed-length spectrograms, so they get padded to a batch tensor; label sequences are variable-length, so they get padded with a mask value the loss function ignores. Get this wrong and the model silently learns to predict padding.

Training arguments then set the shape of the run. Batch size, gradient accumulation, learning rate, warm-up, mixed precision and gradient checkpointing decide whether the job fits in memory at all. Reducing memory usage mostly means trading batch size against accumulation steps. Sequence-to-sequence training also needs generation enabled at evaluation time, since WER has to be computed on decoded text rather than on logits. The dependency list is short: import torch, Transformers, Datasets, and an evaluation library for the metric.

Define evaluation metrics, then begin fine-tuning

Word error rate is the default evaluation metric for ASR, and you should compute it on both the raw and the normalised text. The published Dhivehi reference run shows why: the same model checkpoint reports orthographic WER around 62% and normalised WER around 13%. Same model weights, same audio, one normalisation decision — a five-fold difference in the headline number.

The walkthrough trains for 500 steps. It reports that the fine-tuned model clearly beats the pre-trained Whisper small checkpoint it started from. The test set is a Common Voice 13 subset with roughly ten hours of labelled Dhivehi data — about seven hours of actual training data once the train and validation splits are combined. Watch validation WER across those steps rather than only at the end. A curve that improves on training loss while validation flattens or rises is overfitting, and on ten hours of data it can happen well before step 500.

How Much Labelled Audio Does Audio Model Fine-Tuning Actually Require?

Most project plans over-budget for volume. The published curves show why: the returns flatten within the first ten hours.

The published curves

A UCLA team — Zhen Wang, Nasir Bhanu Shankar, Mengjie Shi, Kaiwen Zhang and Abeer Alwan — measured Whisper-medium on child speech at 13.10% WER zero-shot. One hour of in-domain labelled audio brought it to 9.56%. Ten hours reached 9.19%. The full 133-hour set reached 8.86%. Their layer-selection paper is worth reading for the adaptation method; the curve itself is the planning artefact.

Diarization behaves similarly at even lower volumes.

‍

‍

Two caveats on that bottom row. The Indonesian adaptation work built its 25-hour training set from synthetic TTS-generated (text-to-speech) speech rather than recorded conversation, so the transfer of that figure to real audio is untested. And no source located for this guide plots DER against annotated hours the way the UCLA table plots WER against them. Three two-point comparisons are not a curve, and anyone quoting one as if it were is extrapolating.

Parameter-efficient methods, and the reason to use them

LoRA-Whisper, presented at Interspeech 2024 by Zheshu Song and colleagues, matched monolingual full fine-tuning while training about 5% of parameters. It uses LoRA (low-rank adaptation), a technique that trains a small set of added weights instead of updating the whole model. Its results without adapters are the ones worth planning around. Fine-tuning only on new-language data nearly tripled WER on the original languages. And merging four languages into a single full fine-tune pushed average WER from 9.19% to 11.68%, against four separate monolingual runs. Adding data does not fix that, since the interference comes from the training distributions themselves.

Two things follow for how you configure a run. Updating all the parameters of a large pre-trained model on a small domain-specific dataset is the fastest route to catastrophic forgetting. Freezing the lower encoder layers and adapting only a subset is usually safer — and the UCLA paper's contribution is automating that layer selection. Parameter-efficient fine-tuning also cuts memory overhead enough to adapt a large checkpoint on hardware that couldn't otherwise hold its optimiser states, which is how whisper-large gets fine-tuned on a single 16GB card.

What limits an audio fine-tuning project is consistently annotated hours, not raw recorded ones. The arithmetic above says you need fewer of them than you feared, and that the composition of the set matters more than its size.

Which Component Should You Fine-Tune First?

Adapt segmentation before speaker embedding, because that choice decides what you should be paying annotators to produce.

Speaker embedding models work on acoustics rather than linguistic content, which is why they transfer across languages and domains with relatively little loss. Segmentation decides when each speaker is active, and that depends on the room, the microphone, and how readily your speakers talk over each other. Mohammed Abu Bhuiyan and colleagues at North South University make the point directly in Bangla-WhisperDiar. Md Shahriar Chowdhury and Ahmed Farhan Chowdhury measured it: segmentation fine-tuning moved DER from 0.405 to 0.257, the single largest gain in their pipeline. Partial fine-tuning on just two files already cut DER by nearly 20 points. The error profile shifted underneath the number, too — missed speech dominated before adaptation, speaker confusion after.

How much weight that convergence carries

Several of these results are independent submissions to the same Kaggle competition, on the same ten annotated Bengali recordings (9.61 hours, 2,612 segments in total). Five teams reaching the same conclusion on one benchmark is genuine corroboration that segmentation is the bottleneck. It is not five independent datasets, though, and the absolute figures are self-reported leaderboard numbers on one domain.

What that means for the labelling brief

If segmentation is what you are adapting, the annotation that pays is boundary placement and speaker-turn marking. Voice-identity labelling can wait. In interface terms, that means timed segments drawn on a waveform, with a speaker tagged per segment, over a timeline that survives export.

The same Chowdhury team compared chunking strategies and found that fixed 30-second cuts introduced an average of 2.3 word-boundary errors per boundary, against 0.4 for silence-aware splitting. Cutting long files on a timer costs accuracy before any annotator even opens the file.

Why Does Transcription Style Change Your WER by 60%?

WER counts tokens, and a disfluency is a token whether or not it carries meaning. So two references over the same recording will score it differently, depending on how much filler each one kept.

The nyra labs decomposition in the introduction is the cleanest version of this. An independent researcher, Fabian Akeret, hit the same wall from a different direction while fine-tuning Whisper for Swiss German. Roughly 64% of his evaluation samples were semantically correct but penalised for convention differences — producing 25.60% measured WER against 13.8% content WER. That is a self-published preprint from a single author, on a task that is partly dialect translation, so the absolute figures do not compare to standard ASR benchmarks. The size of the convention penalty still does.

The ambiguous zone

The failure mode that matters for fine-tuning is an uncontrollable model, not just a slightly worse one. Training data mixed between 15% and 85% verbatim creates what the nyra labs team call an ambiguous zone. The model has learned both policies, has no signal telling it which to apply, and switches between them unpredictably at inference. Below and above that band, the behaviour is at least predictable; inside it, whether a given disfluency survives into the output is effectively random.

So declare the convention before the first annotation assignment and enforce it as a hard constraint, because a corpus cannot be un-mixed cheaply. If you need both behaviours, train a control token rather than hoping the model infers intent from context. A clear task definition, written down before data collection starts, is what keeps the rest of the fine-tuning process from being guesswork.

For the full account of what WER normalisation rewrites before scoring begins, the Kili ASR models guide covers it in depth — including why six systems re-scored under different conventions moved by up to half their reported error. This guide adds only the training-side consequence.

What Belongs in an Audio Annotation Guideline?

A usable guideline specifies considerably more than "transcribe what you hear" — and more than most teams write down.

The DISPLACE challenge organisers at IISc Bengaluru and NITK published a working protocol that serves as a template. Annotators marked target-speaker activity from lapel microphones. They annotated the target speaker's non-speech events, such as laughing, coughing and tongue clicks. They marked prominent background sounds as a separate layer and assigned a language label to every word, to capture code-mixing. And they produced transcripts as a third, separate task, rather than as a by-product of segmentation. The difficulty they name is disambiguating speaker switches from code switches, which needs a documented tie-breaker rather than annotator discretion.

Here is the decision set, with what each one costs when it goes unspecified.

‍

‍

Timing needs its own section

Word boundaries are the weakest link in the timing chain. Base Whisper cross-attention gives 203 ms mean absolute boundary error on read speech, and 568 ms on disfluent speech. Forced alignment degrades to 142–200 ms on disfluent input. Supervised alignment reached 36 ms and 102 ms respectively. A guideline that specifies a collar tolerance without specifying how boundaries are to be placed has left that ambiguity with the annotators, who will each resolve it differently. The Kili speaker diarization guide sets out why a DER figure without a stated protocol carries no information; the same logic applies to your internal targets.

Governance is what separates a corpus from a queue

The AMI Meeting Corpus documentation at Edinburgh describes two artefacts per annotation scheme: a version-controlled coding manual, and a separate reliability document. That reliability document reports the cross-coded sample, the agreement measure used, and where substantive disagreement sits. It is corpus documentation rather than a research finding, so treat it as a template rather than as evidence of an effect. Neither artefact is glamorous. Together, though, they are what let you show a convention was applied, not merely written.

Where Does Human Audio Annotation Actually Go Wrong?

Hongyu Sun and colleagues at Amazon built a taxonomy from 1,000 erroneous transcripts for their HTEC paper, and the ranking is not the one most guidelines are written for. Misheard audio dominates at 50.41%. Missing domain knowledge accounts for 18.37%, spelling for 11.63%, grammar for 11.02%. Convention violations come last, at 8.57%.

That ordering complicates this guide's own emphasis on conventions. Convention errors are the smallest category precisely because they are the cheapest to prevent — and the systemic damage they cause is out of proportion to their frequency.

Humans over-insert; models over-delete

The same paper found an asymmetry. Annotator errors run 37.3% insertions to 27.7% deletions, while the ASR system's errors run 16.3% insertions to 42.1% deletions.

A review pass over corrected model output is therefore hunting a different residual error profile than a review pass over from-scratch transcription. One checklist applied to both will systematically miss one of them — and which one it misses depends on which workflow produced the file.

The bridge from annotation quality to model quality has a number attached: every 1% of additional error in human transcription costs roughly 2% in the WER of the model trained on it. HTEC quotes that ratio from a separate Interspeech 2023 paper by overlapping authors at the same company. Cite the original if you lean on it, and note that neither study evaluated non-English data.

The measurement that does not exist

No inter-annotator agreement figure exists for segment-level speaker-attributed transcription scored jointly across words, speaker and timing. Transcription-side agreement figures exist. Diarization-side figures exist. They are measured on different tasks and cannot be summed into a joint estimate. If you need one for your domain, you will have to run the cross-coding study yourself — that is a project to scope, not a box to tick.

Can You Trust a Model's Pre-Annotations?

Where two independent models agree, yes. A single model's output left uncorrected is a different proposition.

The risk, measured on text

Yevgeni Berzak, Yan Huang, Andrei Barbu, Anna Korhonen and Boris Katz at MIT and Cambridge measured anchoring directly on syntactic annotation. Gold standards built by editing a parser's output scored that same parser 33–49% lower in error than a human-built gold standard of the same sentences. Each of two comparable tools scored best on the reference built from its own output. Human judges preferred the human-built gold standard in 64.32% of disagreements.

Nobody has replicated this with waveforms and speaker segments. Treat the percentages as a mechanism, not an audio finding, and don't quote them as one.

The counterweight

Marie Mikulová and colleagues at Charles University reached the opposite verdict on a comparable task. On dependency syntax, with a high-accuracy parser, annotation accuracy held steady within about half a percentage point, while from-scratch work took roughly 1.7 times longer. The authors credit the pre-annotation for the consistency gain: it kept influencing the annotators toward the same choices. Same mechanism, opposite conclusion — unreviewed single-model pre-annotation is the problem, not pre-annotation itself.

The recipe two domains agree on

‍

‍

The bottom row is the audio corroboration. Prasanth Rangappa and colleagues at Idiap, EPFL and Uniphore found that 7,500 hours of pseudo-labelled call-centre audio gave 12.3% WER. Filtering that down to 100 hours — 1.4% of the data — matched it. The filter that won was inter-model agreement. The primary corpus is a vendor's proprietary data with no ground-truth transcripts, so treat the absolute WERs as internal and the agreement result as the transferable finding.

Across both domains, agreement between independent models is the signal that held up under measurement. Single-model output on its own did not.

What skipping that step looks like

Sadman Tanvir, Md Nafis Sakib, Md Ussalam Ahamed and Hasibul Mustafa Al Mukdho documented the failure candidly in ShobdoSetu. Their training references were an ASR system's own output, left uncorrected. The fine-tuned model scored better on the private test set (15.551% WER) than on the public one (16.751%). They explain why: both reference sets came from the same system the model had learned to imitate, so the metric rewarded the circularity.

A model that mimics your pre-annotator will look excellent against references your pre-annotator wrote — and disappoint the first time a human reads its output. The same team later rationed pseudo-labels deliberately, combining all ten human-labelled files with a single API-labelled file per round to limit their influence. They reached a DER of approximately 0.200, no better than fine-tuning without them at all.

Nobody has run Berzak's experiment on audio annotators. So the inter-model agreement route remains the defensible default, not the proven one.

Annotation infrastructure for a fine-tuning run

The workflow above has a shape:

• import pre-labels from a model pass
• correct against the waveform rather than proofread the text
• keep boundaries and speaker tags on one timeline
• export something a training script can read

Kili's audio modality supports segment-level annotation on MP3, MP4, FLAC and WAV. Each segment gets a speaker tag, segment transcription is a mandatory job, asset-level classification is optional, and pre-labels can be imported from a model pass. The audio labeling documentation covers job setup and export payloads.

Explore Kili for audio →

How Do You Know the Fine-Tune Actually Worked?

Hold data out of everything, including the hyperparameter search.

Held out means held out of the sweep too

The ShobdoSetu team used the same ten recordings for both segmentation fine-tuning and the parameter sweep that followed. That produced 0.19974 DER on the public split and 0.26723 on the private one, a 6.7-point gap. They name their own fix: leave-one-out cross-validation. A held-out set that has already informed a single threshold choice is no longer held out.

Contamination has a more spectacular version. In the Swiss German work, a vanilla Whisper model self-trained on the test set — with zero in-language training data — scored 13.88% and beat every published system on that benchmark. The result reflects test-set contamination, not any command of Swiss German.

Metrics that match the task, and monitoring after ship

WER is the default evaluation metric, and it is the wrong default for several audio tasks. A voice-assistant fine-tune cares about intent and entity accuracy more than about filler words. A diarization fine-tune cares about DER decomposed into missed speech, false alarm and confusion. A medical transcription model cares about the terminology it exists to capture, which a corpus-wide WER will average away. Define evaluation metrics that reflect the specific task before training, and read the error breakdown rather than the aggregate.

The same applies after deployment. Acoustic conditions, speaker populations and vocabulary drift over time, so a model's performance on last quarter's audio says progressively less about this quarter's. Sample production audio, score it against fresh references, and feed the failures back into the next round of annotation — that is how the dataset improves. Close inspection of errors is what tells you which examples to collect next.

The failure mode of this guide's own advice

On the same Bengali pipeline, in-domain WER improved from 21.00% to 16.751% across fine-tuning stages. Out-of-domain, though, the base model scored 44.68%, the intermediate fine-tune improved to 40.92%, and the final fine-tune regressed to 41.13%. Five points of in-domain gain, bought with out-of-domain loss, on a narrow and uncorrected corpus.

Fine-tuning on convenience data teaches the model your data's idiosyncrasies, including its errors. The corrective is stratified, deliberately composed training data, not a different learning rate. One team on that benchmark concluded the honest thing: their fine-tuned ASR did not consistently beat the tuned pretrained configuration, so they shipped the pretrained one.

One plain note on tooling: this is where an annotation platform should carry weight, and doesn't carry all of it yet. Honeypot, consensus and review score are not currently supported on audio projects in beta, so programmatic agreement measurement on speech sits outside the platform for now. If your quality plan depends on consensus scoring, plan to compute it on exported payloads, and ask about the beta roadmap before scoping around it.

Does Audio Model Fine-Tuning Make You the Provider Under the EU AI Act?

It can — and for high-risk systems, the trigger has nothing to do with how much compute you spend.

Under Article 25, a third party, deployer, importer or distributor becomes the provider of a high-risk AI system in one of three cases: they rebrand it, they substantially modify it, or they change its intended purpose so that it becomes high-risk. The original provider is then relieved of its obligations but must cooperate and supply information. In practice, the Article 10 data-governance and annotation-documentation duties land on whoever did the fine-tuning — which is a different party from the one the model card names.

One caution on currency: the Commission's own page carries a notice that the provision has been amended by the Digital Omnibus and that the displayed text is not yet updated. Verify the wording against the Commission's Digital Omnibus material before relying on it in a compliance document.

For general-purpose models the test is different. The Future of Life Institute's summary of provider qualification reports a presumption: a downstream modifier becomes a provider once the modification exceeds roughly one-third of the original training compute. Obligations are limited to the modified part. FLI advocates on AI policy and is summarising Commission guidelines rather than issuing them, so check the source text. Ordinary domain fine-tuning of an ASR or diarization model sits nowhere near that threshold either way. Article 25 is the route that should shape your documentation practice; Article 10, the Article 50 marking duty and the amended timeline are covered in the audio annotation guide linked earlier.

Conclusion

Read together, the evidence describes a sequence rather than a set of tips. Write and version the transcription convention before the first assignment, because a corpus mixed between two policies produces a model with no stable behaviour to ship. Annotate boundaries and speaker turns before voice identity, because segmentation is where in-domain data pays off. Pre-annotate with two independent models and accept only where they agree, because a single model's output trains its own imitator and inflates the metric that would have caught it. Hold data out of the sweep as strictly as out of training. Expect most of the gain in the first ten hours, and budget the rest for consistency.

What that sequence produces is a documented corpus with a coding manual, a reliability report and a declared convention. The model weights are downstream of that artefact and cost an hour of GPU time to regenerate from it, as the Whisper walkthrough shows. The corpus is not cheap to regenerate from the weights — and under Article 25, it is increasingly the thing you have to be able to show.

The open questions in this area are specific enough to name. No published study measures anchoring on audio annotators, so the strongest evidence for the two-model workflow still comes from syntactic annotation. No DER-versus-annotated-hours curve exists, only scattered two-point comparisons. No inter-annotator agreement figure exists for speaker-attributed transcription scored jointly. And the strongest single finding in this guide, the 60% style decomposition, has been measured by one team with a product in the category.

None of those gaps closes with a better base model. Every one of them closes with reference data built on real audio under a declared convention, measured rather than assumed — and that work sits with whoever owns the recordings.

Resources

Fine-Tuning Methods and Data Efficiency

• Gumbel-BEARD: Automatic Layer Selection for Self-Supervised Adaptation of Whisper in Low-Resource Domains (Wang, Shankar, Shi, Zhang & Alwan, UCLA, 2026) – the WER-versus-hours curve on child speech; preprint
  • https://arxiv.org/abs/2606.11429
• LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASR (Song, Zhuo, Yang, Ma, Zhang & Chen, Interspeech 2024) – adapter parity, catastrophic forgetting and language interference
  • https://arxiv.org/abs/2406.06619
• Efficient Data Selection for Domain Adaptation of ASR Using Pseudo-Labels and Multi-Stage Filtering (Rangappa et al., Idiap / Uniphore / EPFL, 2025) – inter-model agreement as a filter; vendor-proprietary corpus, preprint
  • https://arxiv.org/abs/2506.03681

Whisper Fine-Tuning Mechanics

• Fine-Tuning the ASR Model (Hugging Face Audio Course, Chapter 5) – feature extractor, data collator, training arguments, WER evaluation and the Dhivehi Common Voice 13 run on a T4 GPU
  • https://huggingface.co/learn/audio-course/chapter5/fine-tuning

Diarization Adaptation

• Domain Adaptation of the pyannote Diarization Pipeline for Conversational Indonesian Audio (Prasetyo, Putra, Ilmi & Azizah, 2026) – DER gains from small adaptation sets; the 25-hour set is synthetic TTS speech; preprint
  • https://arxiv.org/abs/2601.03684
• Bangla-WhisperDiar: Fine-Tuning Whisper and PyAnnote for Bangla Long-Form Speech Recognition and Speaker Diarization (Bhuiyan et al., North South University, 2026) – which component to adapt and why; preprint
  • https://arxiv.org/abs/2605.08214
• Robust Long-Form Bangla Speech Processing (Chowdhury & Chowdhury, 2026) – segmentation as the dominant bottleneck; chunking-strategy costs; preprint
  • https://arxiv.org/abs/2602.21741
• ShobdoSetu: A Data-Centric Framework for Bengali Long-Form Speech Recognition and Speaker Diarization (Tanvir, Sakib, Ahamed & Mukdho, 2026) – circular pre-annotation and evaluation contamination, documented by the team itself; one competition, self-reported; preprint
  • https://arxiv.org/abs/2603.19256

Transcription Convention and Style

• Transcription Policy as a Latent Variable: Activating Controllable Verbatim ASR with Word-Level Timing (Wagner, Zusag & Thallinger, nyra labs, 2026) – the style decomposition, the ambiguous zone and boundary-error figures; commercially interested author affiliation; preprint
  • https://arxiv.org/abs/2607.18934
• Subtitle-Aligned Fine-Tuning of Whisper for Swiss German ASR (Akeret, 2026) – convention penalty and benchmark contamination; single independent researcher, preprint
  • https://arxiv.org/abs/2606.07608

Annotation Guidelines and Governance

• The DISPLACE Challenge 2023: Diarization of Speaker and Language in Conversational Environments (Baghel et al., IISc Bengaluru / NITK, Interspeech 2023) – a published protocol for code-mixed conversational audio
  • https://arxiv.org/abs/2303.00830
• AMI Corpus: Annotation (University of Edinburgh, n.d.) – coding manual and reliability document as standing artefacts; corpus documentation, not a finding
  • https://groups.inf.ed.ac.uk/ami/corpus/annotation.shtml

Annotation Error and Pre-Annotation Bias

• HTEC: Human Transcription Error Correction (Sun, Gao, Wu, Fang, Cao & Du, Amazon, 2023) – error taxonomy, insertion/deletion asymmetry, the 1%-to-2% ratio quoted from Gao et al.; internal dataset, preprint
  • https://arxiv.org/abs/2309.10089
• Anchoring and Agreement in Syntactic Annotations (Berzak, Huang, Barbu, Korhonen & Katz, MIT / Cambridge, 2016) – anchoring measured, and the two-model mitigation; syntactic annotation, no audio replication
  • https://arxiv.org/abs/1605.04481
• Quality and Efficiency of Manual Annotation: Pre-Annotation Bias (Mikulová, Straka, Štěpánek, Štěpánková & Hajič, Charles University, 2022) – the counterweight result on consistency and speed; syntactic annotation
  • https://arxiv.org/abs/2306.09307

Regulation

• EU AI Act, Article 25: Responsibilities Along the AI Value Chain (European Commission, AI Act Service Desk) – when fine-tuning moves provider status; page carries a Digital Omnibus amendment notice
  • https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-25
• Providers of General-Purpose AI Models: What We Know About Who Will Qualify (Future of Life Institute, 2025) – the compute-share threshold; advocacy organisation summarising Commission guidelines
  • https://artificialintelligenceact.eu/providers-of-general-purpose-ai-models-what-we-know-about-who-will-qualify/

Related Reading from Kili

• Audio Annotation Guide: Building Speech Ground Truth You Can Verify (2026) – annotation types, per-hour costs, quality control and the EU AI Act timeline
  • https://kili-technology.com/blog/audio-annotation
• ASR Models Guide: Word Error Rate, Benchmarks and Failure Modes [2026] – what WER forgives before counting, and the benchmark-to-production gap
  • https://kili-technology.com/blog/asr-models-guide-word-error-rate-benchmarks-and-failure-modes-2026
• Speaker Diarization Models Guide: Benchmarks and Failure Modes [2026] – collars, protocols, overlap, speaker-count effects and model licensing
  • https://kili-technology.com/blog/speaker-diarization-models-guide-benchmarks-and-failure-modes-2026
• Labeling Audio Assets (Kili documentation) – job setup, supported formats and export payloads for the audio modality
  • https://docs.kili-technology.com/docs/labeling-audio-assets

Frequently Asked Questions

What is fine-tuning an audio model?

Fine-tuning continues training a pre-trained audio model on a smaller, domain-specific dataset, so it adapts to your vocabulary, speakers and recording conditions. It reuses the pre-trained weights instead of training from scratch. That's why ten hours of labelled audio can produce a usable speech-to-text model where a from-scratch run would need thousands.

How much labelled audio do you need to fine-tune an ASR model?

Less than most plans assume. One published curve on child speech moved Whisper-medium from 13.10% to 9.56% WER on just one hour of in-domain audio, with 133 hours reaching only 8.86%. Most of the available gain arrives early, so budget for consistency across the set rather than for volume.

How long does it take to fine-tune Whisper?

The reference Hugging Face walkthrough completes a full run on roughly ten hours of Common Voice 13 data in under an hour, on a free-tier 16GB T4 GPU, training for 500 steps. Larger checkpoints need parameter-efficient methods to fit the same hardware. Producing the annotated hours takes far longer than the training run does.

What data preprocessing does audio fine-tuning require?

Resample every audio file to the model's expected sampling rate — 16kHz for Whisper. Then normalise loudness, segment long recordings into consistent chunks on silence rather than on a timer, and convert each audio array to the feature representation the model expects. The Whisper feature extractor handles that conversion, producing log-mel input features. Transcripts are tokenized separately and padded by a data collator that treats input features and labels differently.

Should you fine-tune the ASR model or the diarization model first?

Fine-tune segmentation first. It decides when each speaker is active, which is a property of your recording conditions, while speaker embedding models work on acoustics and transfer across domains with less loss. Segmentation fine-tuning produced the largest single DER gain in several published pipelines.

Should you fine-tune all the model's layers?

Usually not. Updating all the parameters of a large pre-trained model on a small dataset invites catastrophic forgetting — LoRA-Whisper measured this as a near-tripling of WER on the original languages. Freezing lower layers, or using parameter-efficient fine-tuning, preserves the pre-trained knowledge and reduces memory usage at the same time.

What is the difference between verbatim and intended transcription, and does it matter?

Verbatim transcripts record false starts, repetitions and filler; intended transcripts record what the speaker meant. On identical audio the two disagree by up to 12.1% WER. Mixing them in one training corpus produces a model that emits disfluencies unpredictably, so the convention has to be declared before annotation begins.

Is it safe to fine-tune on ASR-generated pseudo-labels?

Not from a single system left uncorrected. One documented case produced a model that imitated the reference generator and scored better on the harder test split, because both reference sets came from the same system. Filtering on agreement between independent models is the mitigation with evidence behind it.

Can Whisper be fine-tuned for low-resource languages?

Yes, and that is one of its better use cases. The published Dhivehi walkthrough builds on Whisper's multilingual pre-training with about seven hours of training data — far too little to train an acoustic model from scratch, but enough to adapt one. The same approach extends to new languages and dialects wherever a small labelled set exists.

Does fine-tuning a speech model make you a provider under the EU AI Act?

It can. Article 25 moves provider status to anyone who substantially modifies a high-risk system or changes its intended purpose into a high-risk use, which brings Article 10 data-governance duties with it. The provision has been amended by the Digital Omnibus, so check the current text.

Why did fine-tuning make the model worse on other data?

Fine-tuning on a narrow corpus teaches the model that corpus's idiosyncrasies, including its errors. One documented pipeline improved in-domain WER by five points while out-of-domain performance regressed. Stratified, deliberately composed training data is the corrective, not a different learning rate.