Speaker Diarization Models Guide: Benchmarks and Failure Modes [2026]

Introduction

Diarization error rate, or DER, is the number this field argues about. It sums three quantities: speech the system missed, non-speech it labelled as speech, and speech it attributed to the wrong person. Divide that sum by the total duration of speech in the reference annotation and you have DER. A 12% DER means the system got twelve seconds wrong for every hundred seconds of speech in the reference.

Before any of those errors are counted, standard scoring protocols inherited from NIST practice discard 250 ms on either side of every reference speaker turn: half a second per boundary, removed from the calculation. That tolerance is called the forgiveness collar, and it exists because the reference annotation itself cannot place boundaries precisely. NIST's estimate was that 250 ms accounts for the labelling differences between human annotators.

The ground truth that speaker diarization models are measured against has a documented resolution limit, and the metric was designed around it. Most of the distance between a low DER on a leaderboard and the transcript your compliance team is complaining about starts there.

Diarization in 2026 is better than it was in 2023. Hybrid systems built on self-supervised speech encoders have pushed open benchmark results down substantially; streaming architectures made low-latency operation practical. What has not changed is that benchmark corpora control for clean channels, known speaker counts, modest overlap and forgiving boundaries, and real recorded conversations violate all four. This guide covers what the main model families do, which speaker diarization models are actually in production use, how DER is constructed, where it breaks on real call audio, and what human review can still fix.

What Is Speaker Diarization, and What Does It Actually Output?

Speaker diarization, also written speaker diarisation in British usage, answers "who spoke when." It partitions an audio stream into speaker-homogeneous segments, each with a start time, an end time, and a speaker index, conventionally serialised as an RTTM segment list.

The labels are relative, not identities. A diarization system returns SPEAKER_00 and SPEAKER_01, and nothing in the output tells you which one is the agent and which is the customer. Mapping those indices onto a real speaker identity is a separate problem with separate requirements.

The reason to perform speaker diarization at all is that most useful things you can do with recorded human speech need to know who said them. Media archives are indexed by speaker so that broadcast news and conference footage become searchable by contributor. Medical consultations are transcribed with statements attributed to the clinician or the patient. Legal proceedings depend on the same attribution, and so does any review of a call between an agent and a customer. In each case the final transcript is only as useful as its speaker labels.

NVIDIA's streaming Sortformer emits a T × S matrix of per-frame speaker activity probabilities over 80 ms frames, and that matrix is what the model produces. Everything downstream, including the tidy boundaries in your RTTM file, comes from thresholding and smoothing it. The crisp edges in the output are a rendering choice applied to a probabilistic surface.

Diarization is also not transcription, and the two get conflated constantly. Automatic speech recognition tells you what words were said; diarization tells you how the audio divides between distinct speakers. Whisper does not diarize. Speech-to-text pipelines that appear to do both are running two models and joining their outputs.

How Do the Main Speaker Diarization Approaches Differ?

Four method families are in active use, and each fails in a way that follows from its architecture.

‍

The clustering pipeline is the shape most write-ups still describe, and it remains widely deployed, but it carries a hard architectural limit: each segment receives exactly one speaker label, so two people talking at once cannot be represented at all. That limitation is why end-to-end neural diarization was proposed. Fujita and colleagues reformulated the task as multi-label classification with permutation-invariant training, which optimises the error rate directly and can emit two speakers in the same frame.

The five neural building blocks of a clustering pipeline

Anyone debugging a diarization pipeline needs to know which stage produced the error, because the stages fail differently.

Voice activity detection decides which parts of the audio content contain human speech at all, and every later stage inherits whatever it gets wrong. Speaker segmentation then cuts the detected speech into short audio segments, splitting at change points where the voice appears to switch. Feature extraction converts each of those speech segments into a mathematical representation of its acoustic properties, historically an x-vector and now more often a neural speaker embedding from a deep learning encoder. Speaker clustering groups those representations by acoustic similarity, using hierarchical clustering in older stacks and variational Bayesian HMM clustering over x-vector sequences in current ones. A final re-segmentation pass, optional in most implementations, refines the boundaries between speaker turns so transitions land in sensible places rather than mid-word.

Signal processing conventions shape all five. Most systems ingest 16 kHz mono audio and resample anything else to match, which is one reason narrowband telephone audio behaves differently from the meeting recordings these pipelines were tuned on.

EEND traded one constraint for another. A fixed output dimension means a fixed maximum speaker count. The standard fix, encoder-decoder based attractors, extends EEND to an unknown number of speakers. Hybrid systems sidestep the problem differently, running neural speaker diarization locally on short windows and clustering the resulting speaker embeddings across the whole recording.

Hybrid systems currently produce the strongest open benchmark results. Most listicles still describe diarization as a clustering pipeline and nothing else.

Which Speaker Diarization Models Are People Actually Using in 2026?

Two model families cover most serious open deployments, sitting alongside a set of general-purpose toolkits that supply components rather than turnkey pipelines.

‍

‍

DiariZen's model weights are released under CC-BY-NC-4.0, and the maintainers are unusually explicit about why. The restriction is inherited from the training data. RAMC, MSDWild and DIHARD-3 prohibit commercial use outright, while AISHELL-4 and AliMeeting carry share-alike terms that require permitting it, and BUT resolved the conflict by applying the most restrictive condition and telling downstream users they assume the legal risk if they ship it commercially. The source code is MIT. The weights are research-only.

A model's licence is set by the provenance of the data it was trained on. Anyone planning a production deployment needs to read the dataset terms, not only the model card.

On the NVIDIA side the trap runs the other way. The original Sortformer checkpoint carries CC-BY-NC-4.0 and is therefore non-commercial, while the streaming v2 checkpoint is CC-BY-4.0. Teams that pinned v1 two years ago should check what they shipped.

If you are choosing an architecture, the underlying DiariZen research sets out the trade-offs. The paper Han and colleagues presented at the 2025 IEEE International Conference on Acoustics, Speech and Signal Processing showed that a pre-trained WavLM encoder sharply reduces the volume of training data neural networks need for diarization, and that simulated data, long treated as necessary for EEND training, did not help in that setup. A follow-up on structured pruning of self-supervised models cut WavLM Large from 316.6M parameters to 63.3M at 80% sparsity without losing accuracy, provided the encoder was fine-tuned on the diarization task before pruning rather than after.

"Whisper diarization" has entered the vocabulary as a category error, and WhisperX is where it comes from. Bain and colleagues built an orchestrator: Whisper produces utterance-level timestamps that can be seconds off, wav2vec2 forced alignment recovers word-level timing, a separate diarization model supplies the speaker segmentation, and a joining step assigns words to speakers. The repository pins a specific diarization dependency, and whichever model occupies that slot determines both the error profile and the licence you inherit. Check it before you build on it.

For toolkits, SpeechBrain supplies ECAPA-TDNN embeddings that appear inside many diarization stacks, and Alibaba's 3D-Speaker toolkit adds semantic and visual modules on top of acoustic embeddings. Neither is a turnkey diarizer, and describing them as one produces a comparison that falls apart on contact.

For regulated buyers, one property outranks any DER figure: these models run offline. Sortformer runs locally through NeMo, DiariZen loads pre-trained weights from disk, and nothing in either pipeline requires an outbound connection at inference. Whether audio ever leaves your network is a deployment decision you control.

How Is Diarization Error Rate Measured, and What Does DER Hide?

Return to the definition: missed speech plus false alarm speech plus speaker confusion, over total reference speech time. That single evaluation metric is how nearly all speaker diarization systems get compared. The complication is that two parameters applied before scoring change the answer by more than a full model generation's worth of progress.

The first is the collar, described above. The second is whether overlapping speech is scored at all. NIST-era practice excluded it. DIHARD broke with that convention, applying no forgiveness collar and scoring overlap, which is why DIHARD numbers and NIST-style numbers are different quantities that happen to share a name.

Put the two protocols side by side on the same system family and the gap is obvious. DiariZen's maintainers publish under the least forgiving setup available, no collar at all, and report 9.1% on VoxConverse and 14.5% on DIHARD 3 full. An independent evaluation from ETH Zürich applying a 0.25 s collar with overlap scored puts DiariZen at 5.2% on VoxConverse. Different checkpoints within the same family, same corpus, and the protocol accounts for most of a result that nearly halves. The same pattern appears in NVIDIA's own reporting, where DIHARD III figures are given at a 0 s collar and CALLHOME figures at 0.25 s, so the two cannot be read against each other even though they sit in the same table.

A DER number without a stated protocol is not information. If a paper, vendor or blog post gives you a percentage without saying whether a collar was applied and whether overlap was scored, the number cannot be compared to anything.

The harder version of this problem is the reference itself. A team at Imperial College London had thirteen human reviewers diarize the same five-minute AMI extract by ear. Mean DER against the reference was 15.60% with no collar, standard deviation 1.53. Eight of the thirteen identified between 35 and 38 segments, and their DERs still differed by as much as 5.86 percentage points. Applying a 250 ms collar lowered every reviewer's DER to a mean of 8.95% while increasing the spread, which the authors read as evidence that reviewer disagreement was not concentrated at boundaries at all. Their conclusion was blunt: "The use of forgiveness collars is shown to be unhelpful."

Then they scored the same reviews against four differently-constructed ground truths for the same audio. Mean DER moved from 8.95% to 11.93%. A three-point swing, with no model involved, purely from which defensible version of the truth you pick.

Two further metrics exist because DER hides things. JER was introduced by DIHARD so that participants who speak for most of a meeting cannot dominate the score. cpWER measures transcription and speaker attribution jointly; the gap between cpWER and plain WER isolates the cost of attribution error specifically.

The standard metric leaves other things out entirely. DER says nothing about transcription accuracy, since it never looks at words. It also says nothing about whether the speaker labels a system emits line up with the ones anything downstream expects, because the scoring procedure finds the best possible mapping between hypothesis and reference labels before it counts errors. A system can post a respectable diarization error rate and still hand your pipeline a SPEAKER_00 that means something different in every file.

What Breaks When You Run These Models on Real Recorded Conversations?

Two findings run against expectation, and both come from the systems' own builders or from independent evaluation.

The first is a speaker-count cliff. NVIDIA's streaming Sortformer is architecturally capped at four speakers, and the model card states that performance degrades on recordings with five and more. The published numbers show how much: on DIHARD III eval with overlap included and no collar, 13.24% DER for sessions with one to four speakers against 42.56% for sessions with five to nine. On CALLHOME part 2 with a 0.25 s collar, 6.57% at two speakers rising to 28.74% at six. Cold transfers, supervisors joining mid-call and three-way conferences are exactly the case where a recording crosses that line partway through.

The second is boundary precision. The ETH Zürich evaluation ran five systems across 196.6 hours in five languages and found missed speech to be the largest DER component for nearly every model. Short segments were not what went missing: those under 0.5 s accounted for under 5% of errors, while the average missed segment ran around 350 ms. Turn edges are fuzzy everywhere, and the collar is precisely what hides fuzzy turn edges. The best-scoring system in that evaluation was a commercial API whose implementation is not publicly documented, so the ranking is only reproducible for the open models, of which DiariZen led at 13.3% average DER. The authors' summary of the field, after all that measurement, was that "it remains an unsolved problem."

The rest of the failure surface:

‍

‍

Two challenges dominate any error analysis of a diarization run. The first is overlapping speech, which the table above covers. The second is acoustic similarity between participants. When different voices share pitch, accent and channel conditions, the embeddings the clustering stage works from sit close together, turns get merged as though they came from the same speaker, and the system's estimate of how many speakers are present drifts with them. That is speaker confusion, and it is the DER component least helped by a better voice activity detector, because nothing upstream can separate voices the encoder does not distinguish. Family members, colleagues from one region, and same-gender pairs on a narrowband line are the ordinary cases.

Corpus provenance matters more than it looks. VoxConverse, the corpus with the lowest DER in almost every table, is built from YouTube videos of debates and interviews where speakers are miked and take turns. AMI is meeting recordings from headsets and arrays; CALLHOME is decades-old landline audio. No academic evaluation of diarization on modern contact-centre recordings at scale appears in the public literature at all, so every number in this guide comes from one of those three worlds. Treating a result on any of them as a prediction about a single audio recording from your own traffic is the most common mistake in vendor selection.

Overlap mismatch runs deeper than the table suggests. Fujita and colleagues reported that the real telephone data used to train their EEND system contained just 3.7% overlapped speech, against 13.0% in the CALLHOME test set. A model tuned on one overlap regime is mismatched on another before it sees a single production call. And overlap is not evenly shaped: analysis of AMI headset audio puts it at 81% single-speaker, 15% two-speaker and roughly 4% three-or-more, so the two-speaker case accounts for about three-quarters of all overlap regions.

The single-label floor is a hard bound. No amount of tuning moves it, and Landini and colleagues showed that on VoxCeleb development data containing 2.9% overlap, a system emitting one speaker at a time cannot score below 2.3% missed speech DER even with perfect voice activity detection and zero speaker confusion.

The fairness picture is thinner than anyone should be comfortable with. The only diarization-specific fairness protocol we found measured correct single-speaker detection at 90.98% for US accents, 83.48% for African accents and 68.37% for Singaporean, with male voices over-split at 10.49% against 5.84% for female. The authors disclose full financial support from a commercial video-indexing company, and state three limits themselves: the protocol runs on single-speaker utterances rather than conversations, results depend on the VAD and clustering chosen, and a system that always predicted exactly one speaker would score a perfect fairness rate. Treat those numbers as evidence of a measurement gap. The ASR-side literature is far more developed, with Koenecke and colleagues in PNAS documenting error rates roughly double for Black speakers against white speakers, and the diarization equivalent for multi-party conversation simply does not exist. The corpora lack the metadata that would make it possible.

How Do Diarization Errors Compound Downstream?

Attribution error does more damage than its share of the percentage suggests. A sentence assigned to the wrong speaker changes what the transcript says happened. In a dispute reconstruction, "I accept those terms" attributed to the wrong party is a different document.

This is what cpWER and its relatives were built to capture. Because cpWER scores transcription and attribution jointly, the delta between cpWER and plain WER is the measurable cost of getting the speakers wrong. Related variants split the problem further: tcpWER adds a time constraint, ORC-WER and MIMO-WER score speaker-agnostically, and WDER measures attribution at word level.

In some sectors the transcript is evidence, which raises the stakes. MiFID II Article 16(7) requires firms to record relevant client conversations, with retention of at least five years and up to seven on request; the UK equivalent sits in FCA SYSC 10A. Where a recording is retained for that purpose, speaker attribution is part of what makes it usable.

The same logic applies wherever a call recording feeds a decision. Service-quality review depends on knowing which turns belong to the agent. Evaluating a voice agent against recordings of human conversations depends on the same thing, and an evaluation built on mis-attributed turns measures the diarizer as much as the model under test.

What Can You Actually Do About Diarization Error?

Start with the constraint the whole field operates under: reference data for diarization is expensive, slow, and partly automated even at the top of the research stack. The Third DIHARD Challenge evaluation plan is unusually candid about this. Their segmentation spec asked for boundaries within 10 ms of the true boundary, splitting on pauses over 200 ms. Producing it required highly skilled annotators working with multiple spectrogram displays, ran at "real time rates typically greater than 15X and sometimes exceeding 30X," and many annotators could not perform the task even after training. For DIHARD III the organisers abandoned fully manual segmentation in favour of forced alignment.

So human annotation of diarization is hard. The Imperial study says something more specific. When their reviewers were given the segment boundaries and only had to say who spoke, mean DER collapsed from 15.60% to 1.41%. Given ground-truth speech activity but not the turn structure, 4.49%. Humans are excellent at telling voices apart and poor at deciding where a turn starts and what counts as speech. The same split shows up in the DIHARD III challenge itself, where the track with ground-truth speech activity was won at 13.45% against 19.37% for the track that had to find speech unaided.

That result has a direct read for tooling. It rests on one five-minute extract, so do not over-generalise, but it fits the other data point available: VoxConverse's annotators, working with video and correcting the output of an automatic pipeline rather than starting from silence, differed by around 1% DER on a one-hour subset. Roughly 1% between annotators correcting a machine, roughly 15.6% between annotators working from audio alone. Two studies, neither on telephone audio, and the contrast between their conditions is more informative than either number alone.

Correction also demonstrably works on the output side. Researchers at Worcester Polytechnic Institute reduced DER on the AMI headset test set by 31.99% relative and speaker error by 52.68% relative over a streaming baseline through lightweight human corrective feedback, closing roughly 70% of the gap to the oracle bound. They also observed that ASR segments frequently merge multiple speakers, which is precisely the seam where attribution errors originate.

Automated diarization degrades under exactly the conditions real recorded conversations present, so production systems that need reliable attribution end up putting humans in the loop. That makes correcting diarization output a data quality problem with its own requirements, and the evidence above sets most of them.

Reviewers need the existing segmentation and the waveform in front of them, because correction is where they are reliable and construction is where they are not. Agreement between reviewers on boundary placement has to be measured rather than assumed, since the same audio scored against four defensible references moved three points. And when the recordings are client calls retained under a regulatory obligation, the review environment inherits the retention and residency constraints of the recordings themselves, which is why on-premise deployment tends to be the first question asked.

This is the work Kili Technology is built for: multi-stage review with configurable quality gates, consensus measurement across reviewers, audit trails that survive an external inspection, and deployment on your own infrastructure when the audio cannot leave it. Diarization correction is one instance of a general pattern, which is that human judgments feeding a model need the same quality controls as any other production data.

Conclusion

The honest summary of speaker diarization in 2026 is that the models improved and the measurement did not keep up. Hybrid architectures built on self-supervised encoders produce better open results than clustering pipelines did two years ago, streaming diarization became practical, and licences shifted underneath teams who were not watching. None of that changes the fact that a published DER describes a specific corpus under a specific protocol, and that the reference it was scored against was produced by people who disagree with each other by several percentage points.

The licence terms carry a second lesson. DiariZen's weights are research-only because three of its training corpora forbid commercial use, and that restriction propagates to every system built on top of them. Data provenance determines what you are allowed to ship, which means the governance of a training set outlives the training run by a long way.

The practical consequence is a change in what you should ask for. Not "what is your DER" but "on what audio, under what collar, with overlap scored or not, and against whose reference" — and then, because the answer will not transfer to your traffic, a measurement on your own audio data with your own speaker distribution and your own channel conditions.

The field's own gaps point the same direction. There is no public DER on modern contact-centre audio, no diarization fairness study on multi-party conversation, and no measured curve for long-form drift. Those absences will not be filled by a better checkpoint. They will be filled by organisations that build reference data on their own audio and hold it to a standard they can defend, which is a data operations problem long before it is a modelling one.

Resources

Metrics and Evaluation Protocols

• DIHARD I task overview – the challenge that dropped the collar and scored overlapping speech
  • https://dihardchallenge.github.io/dihard1/overview.html
• Third DIHARD Challenge Evaluation Plan (Ryant et al., 2020) – annotation cost, segmentation spec, and the move to forced alignment
  • https://arxiv.org/pdf/2006.05815
• Word Error Rate Definitions and Algorithms for Long-Form Multi-talker Speech Recognition (von Neumann, Boeddeker et al., 2025) – cpWER and related attribution metrics
  • https://arxiv.org/abs/2508.02112

Open Models, Toolkits and Licences

• DiariZen (BUT Speech@FIT) – benchmark table under a no-collar protocol; MIT code, CC-BY-NC-4.0 weights, with the data-provenance rationale
  • https://github.com/BUTSpeechFIT/DiariZen
• Leveraging Self-Supervised Learning for Speaker Diarization (Han et al., ICASSP 2025) – the WavLM + Conformer result behind DiariZen
  • https://arxiv.org/pdf/2409.09408
• Efficient and Generalizable Speaker Diarization via Structured Pruning of Self-Supervised Models (Han et al., 2025) – 80% sparsity without accuracy loss
  • https://arxiv.org/pdf/2506.18623
• NVIDIA diar_sortformer_4spk-v1 – non-commercial CC-BY-NC-4.0 licence
  • https://huggingface.co/nvidia/diar_sortformer_4spk-v1/blob/main/README.md
• NVIDIA diar_streaming_sortformer_4spk-v2 – streaming checkpoint and the four-speaker limitation
  • https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2
• SpeechBrain: A General-Purpose Speech Toolkit (Ravanelli et al., 2021) – ECAPA-TDNN embeddings used across diarization stacks
  • https://arxiv.org/abs/2106.04624
• 3D-Speaker-Toolkit (Chen et al., Alibaba) – multimodal speaker verification and diarization
  • https://arxiv.org/pdf/2403.19971
• WhisperX repository – current pipeline composition and diarization dependency
  • https://github.com/m-bain/whisperx

Method Papers

• End-to-End Neural Diarization: Reformulating Speaker Diarization as Simple Multi-Label Classification (Fujita et al., 2020)
  • https://arxiv.org/abs/2003.02966
• End-to-End Neural Speaker Diarization with Self-Attention (Fujita et al., 2019) – training/test overlap mismatch figures
  • https://arxiv.org/pdf/1909.06247
• End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors (Horiguchi et al., 2020)
  • https://arxiv.org/abs/2005.09921
• End-to-End Speaker Segmentation for Overlap-Aware Resegmentation (Bredin & Laurent, 2021) – AMI overlap structure
  • https://arxiv.org/pdf/1910.11646
• WhisperX: Time-Accurate Speech Transcription of Long-Form Audio (Bain et al., Interspeech 2023)
  • https://www.isca-archive.org/interspeech_2023/bain23_interspeech.html

Evaluation Studies and Failure Analysis

• Benchmarking Diarization Models (Lanzendörfer et al., ETH Zürich, 2025) – five systems, 196.6 hours, five languages; preprint
  • https://arxiv.org/html/2509.26177v1
• Analysis of the BUT Diarization System for VoxConverse Challenge (Landini et al., 2020) – the single-label DER floor
  • https://arxiv.org/pdf/2010.11718
• Towards Measuring and Scoring Speaker Diarization Fairness (Tevissen et al., 2023) – industry-funded; single-speaker utterances only
  • https://arxiv.org/pdf/2302.09991
• Racial Disparities in Automated Speech Recognition (Koenecke et al., PNAS, 2020) – ASR-side disparity, not transferable to diarization
  • https://5harad.com/papers/asr-disparities.pdf

Human Annotation and Correction

• Studying Human-Based Speaker Diarization and Comparing to State-of-the-Art Systems (McKnight, Hogg, Neo & Naylor, APSIPA ASC 2022)
  • http://www.apsipa.org/proceedings/2022/APSIPA%202022/TuPM1-3/1570839565.pdf
• Spot the Conversation: Speaker Diarisation in the Wild (Chung et al., 2020) – VoxConverse annotation guidelines and inter-annotator agreement
  • https://arxiv.org/pdf/2007.01216
• Interactive In-Meeting Speaker Correction with Human Feedback (He et al., WPI, 2026) – human correction results; preprint
  • https://arxiv.org/html/2509.18377v2

Regulatory

• MiFID II Article 16(7) and FCA SYSC 10A – call recording and retention requirements
  • https://www.lexisnexis.com/en-gb/legal/guidance/fca-telephone-electronic-communication-recording-requirements

‍