Inter-Rater Reliability for AI Evaluation: Agreement, Gold Answers and Thresholds

Introduction

Kili's ImpactBench data story looked at one LLM judge (a large language model used to grade other models' outputs) scoring 48,540 conversations. The headline validation number was a Cohen's kappa of 0.667 between the judge and the majority expert verdict. Is that good? On the most quoted scale it is "substantial". On a stricter clinical scale it is only moderate. By the rule most content analysts follow, it sits exactly on the floor for tentative conclusions. Same number, three verdicts.

The ImpactBench paper used three different agreement statistics. Cohen's kappa compared the judge with the experts. Fleiss' kappa compared the judge with itself across three runs (0.731). Krippendorff's alpha compared the experts with each other (0.121 to 0.538). The experts agreed with each other less than the judge agreed with them.

Inter-rater reliability is the degree to which independent raters, given the same items and the same guideline, assign the same labels. In AI evaluation the raters are domain experts scoring model outputs, annotators building a golden dataset, or an LLM judge standing in for both. Their labels may become an answer key, a reward signal or a validation set for a judge. Before that happens, you need to know two things: whether they agree with each other, and whether they are right. Those are different questions with different measurements. This guide covers both, the statistics that go with each, the thresholds people use and what to do when you fall short.

What Is Inter-Rater Reliability, and Why Does It Matter for AI Evaluation?

Every evaluation that uses human judgment assumes the judgment is reproducible. If two clinicians would grade the same model answer differently, a model's score depends on which clinician it got. If your gold labels come from raters who agree only by luck, an LLM judge validated against them is validated against noise.

Artstein and Poesio's survey of agreement in computational linguistics puts the logic plainly. If raters produce consistent results, you can infer they share an understanding of the guideline. If they don't, either some of them are wrong or the guideline is unfit for the data. They add a warning: "achieving good agreement cannot ensure validity." Two raters can share the same mistake.

So agreement and accuracy need separate measurements. Inter-rater reliability (also called inter-annotator agreement) tells you the labels are consistent. Comparison against a trusted reference, a gold answer, tells you whether they are correct.

Why Isn't Percent Agreement Enough?

Percent agreement is the share of items on which raters gave the same label. It is easy to compute and easy to explain, and it overstates reliability whenever raters could agree by chance.

Take two experts grading 100 model answers as pass or fail.

• In the first set, half the answers are good. They agree on 45 passes and 45 fails, and split on 10. Agreement is 90%.
• In the second set, almost every answer is good. They agree on 90 passes, split on 10, and never agree on a fail. Agreement is still 90%.

The second pair never agreed that anything failed. Raters who marked almost everything "pass" would match 90% of the time without reading the answers. Cohen's 1960 paper introduced kappa to correct for that:

κ = (observed agreement − chance agreement) / (1 − chance agreement)

Chance agreement comes from how often each rater uses each label. In the balanced set, chance agreement is 0.50, so κ = 0.80. In the skewed set, chance agreement is 0.905, so κ ≈ −0.05: slightly worse than chance, at 90% agreement.

This is the first of the two paradoxes Feinstein and Cicchetti described in 1990: high agreement, low kappa, when labels are unevenly distributed. AI evaluation data is often skewed in exactly this way: when most outputs pass a check, "pass" dominates the labels. Their companion paper recommends reporting kappa next to two separate figures: agreement on positive decisions and agreement on negative ones. In the skewed set, positive agreement is 0.95 and negative agreement is zero.

The same issue shows up in LLM judge research. Zheng et al. found human-to-human agreement on pairwise votes for MT-Bench (a set of multi-turn chat questions) of 63% when ties counted and 81% when they didn't, against random baselines of 33% and 50%. Thakur et al. tested 13 judge models and found percent agreement high for nearly all of them, while a chance-corrected coefficient (Scott's π) separated them. Kili's data story Who Judges the Judge? shows raw agreement overstating judge quality by 33 to 41 percentage points on MT-Bench once chance was removed.

Report percent agreement, but never on its own.

Which Agreement Metric Should You Use: Cohen's Kappa, Fleiss' Kappa or Krippendorff's Alpha?

All three are chance-corrected: 1 is perfect agreement, 0 is what chance would produce, and negative values mean systematic disagreement. They differ in how many raters they handle, what kind of labels they accept and how they estimate chance.

‍

‍

Cohen's kappa

Cohen's kappa compares two raters. It estimates chance from each rater's own label distribution, so a rater who uses "fail" more often than the other changes the expected agreement. It suits a fixed pair of experts, or a judge scored against one reference rater, as in ImpactBench.

Fleiss' kappa

Fleiss (1971) extended agreement to many raters: every item gets the same number of ratings, but different items can be rated by different people. That fits crowd or expert pools where work is distributed. Artstein and Poesio note that Fleiss's coefficient generalises Scott's π, which pools all raters' labels to estimate chance, rather than Cohen's kappa. With more than a handful of raters the difference shrinks.

Krippendorff's alpha

Krippendorff's alpha works with any number of raters, any number of categories, missing ratings and any level of measurement: nominal (unordered categories), ordinal (ranked), interval or ratio. It is defined as one minus the ratio of observed to expected disagreement. For AI evaluation that flexibility matters. Rubric scores on a 1 to 5 scale are ordinal, and a near miss (4 against 5) should count less than a full miss (1 against 5). Expert time is scarce, so not every item gets every rater. Artstein and Poesio recommend alpha when labels are not equally distinct from one another. One caution: the distance weights you choose change the result. In one of their examples, alpha ranged from 0.122 to 0.998 on the same data.

What Does Accuracy Against Gold Answers Measure That Agreement Doesn't?

Agreement says your raters are consistent with each other. A gold answer says whether a given rater is right. A gold answer is a label you trust because a senior expert wrote it, a panel adjudicated it, or the answer can be checked. It is also called a reference label or ground truth, and a honeypot when it is mixed into normal work.

The two measurements catch different failures:

• High agreement, low accuracy: raters share a misreading of the guideline, a bias or a shortcut. Agreement looks fine; every gold check fails the same way.
• Low agreement, high accuracy for some raters: one or two people get it right and the rest don't. The gold check tells you who.
• Low agreement, gold answers disputed: the task itself is ambiguous. More training won't fix it; the guideline or the label set needs to change.

Snow et al. (2008) used both in one study. They compared crowd labels with existing expert gold labels across five language tasks, and also compared agreement levels between experts and non-experts. For affect recognition, it took an average of four non-expert labels per item to match the agreement of a single expert.

Gold answers have limits of their own. They are only as good as whoever wrote them, and they go stale as guidelines change. For subjective criteria there may be no single right answer. Guerdan et al. studied items that admit more than one valid rating. Forcing a single gold label on them biased judge validation. Judges chosen that way performed up to 31% worse than judges chosen with multi-label ratings. Kili's guide to building golden datasets covers how to build and maintain the reference set.

What's the Difference Between Consensus and Honeypot?

Consensus measures annotators against each other, and honeypot measures them against a reference you supply. They are the platform versions of the two measurements above.

• Consensus sends the same asset to several annotators independently and scores how much their labels agree. It needs no answer key, so it works on new data and on subjective tasks. A high score means consistency, not correctness.
• Honeypot scores each annotator's label against a reference label you upload for selected assets. It measures individual accuracy, but only on the assets you prepared a reference for, and only as well as that reference is right.

Use consensus to find ambiguous items, unclear guidelines and assets that need adjudication. Use honeypot to qualify new raters, track each rater's accuracy and catch drift. A common setup uses both: consensus on a sample of production items, and honeypots spread through the queue. Kili's earlier overview of data quality metrics covers them alongside IoU and review scores.

What Agreement Threshold Should You Set?

No single number fits every project. The published conventions disagree, and where you set the bar should depend on what the labels are for.

‍

‍

Landis and Koch proposed their bands in 1977, illustrated with a clinical diagnosis example, and they became the best-known convention for describing kappa. McHugh argues that this scale "may be too lenient for health related studies," since it implies a kappa as low as 0.41 might be acceptable. She treats anything under 0.60 as inadequate and notes that many texts recommend at least 80% raw agreement. In Artstein and Poesio's own annotation work, only values above 0.8 produced data of reasonable quality. If a threshold is needed, 0.8 is their pick. They also doubt one cutoff suits every purpose, and recommend reporting the full method instead.

A practical policy for expert evaluation data:

1. Aim for 0.8 on a chance-corrected metric for labels that will become an answer key, a gold set or training data.
2. Treat 0.67 to 0.8 as provisional. Use the labels for exploration, and don't base a release decision on them alone.
3. Below 0.67, don't use the labels as ground truth until you know why raters disagree.
4. Report percent agreement, the metric, the number of raters per item and how ties or "unsure" ratings were handled. Without those, the number can't be compared with anything.

For preference data, Kili's DPO vs RLHF guide shows how published pipelines set agreement thresholds and what share of data they dropped to meet them. Long written answers are harder to score this way. Kili's guide to building an SFT dataset uses overlap scores only for short answers and a review rubric for the rest.

What Should You Do When Agreement Falls Below the Threshold?

A low score starts a diagnosis. Work through it in this order.

Find where the disagreement sits

Break agreement down by label, by item type and by rater. A low overall kappa often hides one class that nobody applies consistently, or one rater who reads the guideline differently. ImpactBench's expert agreement ranged from 0.121 on one metric to 0.538 on another. The low one is where the rubric needs work.

Fix the guideline, then recalibrate

Systematic disagreement often points to an unclear criterion, a missing edge case or two labels that overlap. Rewrite the guideline, add worked examples for the disputed cases, and rerun a calibration round on gold items before going back to production. Kili's quality workflow best practices suggest starting strict (full review, consensus on many assets) and relaxing controls only once agreement stabilises.

Separate rater error from genuine ambiguity

Check the low-agreement items against gold answers. If raters disagree with each other and with the gold, it is a training problem. If they disagree on items where experts themselves would split, the disagreement is information. Aroyo and Welty call the single-correct-answer view "an antiquated ideal of a single correct truth." Plank (2022) argues that genuine label variation carries signal for data, modelling and evaluation, and shouldn't be averaged away as noise. For evaluation, that can mean scoring the model against the distribution of expert ratings, or excluding contested items from the gold set and reporting them separately.

Adjudicate what you keep

Where you need one label, send disagreements to a senior reviewer and record the decision and who made it. A majority vote of two raters isn't adjudication.

Measure the humans before the judge

If you are validating an LLM judge, compute expert-to-expert agreement first. If the experts don't agree with each other, there is no stable target for the judge to agree with.

How Do You Measure Inter-Rater Reliability in Kili?

Kili runs both measurements inside the labeling workflow, while the work is being done.

Consensus is enabled on a labeling step in the Workflow Editor. You set the number of labelers per asset and the coverage, the percentage of assets that go through consensus. Since the 7 October 2026 release (v26.2.12), labelers work on a consensus asset in parallel, and labeling stays blind: no one sees the others' annotations. Once every required label is in, Kili computes a consensus score. At the review step, reviewers compare the labels in Explore, then either validate a consolidated label or send one label back to its author for correction. The 21 October 2026 release (v26.2.13) lets Project Admins and Team Managers add assets to consensus, or remove them, from the Queue page. So far this appears only in the changelog.

Honeypot is enabled on one workflow step per project. You upload reference labels for selected assets through the SDK, and each label submitted at that step is scored against the reference. Project Admins and Team Managers see honeypot scores in Explore, on the Queue page and in the Quality Insights tab of the Analytics page. Quality Insights breaks scores down per class and per labeler.

Two details matter for the statistics above. First, Kili's calculation rules depend on the task. Classification uses the share of labelers choosing each class. Boxes, polygons and segmentation masks use IoU (intersection over union, the overlap between two shapes). Transcription uses the Dice coefficient, a text-overlap score, and named entities use character overlap. They describe direct agreement, with no chance correction. Read them the way you would read percent agreement. Second, when you need kappa or alpha, export the latest label from each author for each asset and compute it on that file.

Neither consensus nor honeypot is available on audio projects; audio is in beta, and its documentation lists quality metrics as not yet supported.

Conclusion: Report the Method With the Number

A kappa of 0.667 can be substantial, moderate or tentative depending on whose scale you use. An agreement statistic measures your raters and your guideline together. It only means something next to the method that produced it: which coefficient, how many raters, how skewed the labels were and what happened to the disagreements.

As LLM judges take over routine scoring, the human labels that validate them become the most important data in the pipeline, and they need the same scrutiny as the judge. Teams that measure expert agreement first, check experts against gold answers separately, and publish both will be able to defend their evaluation numbers. Teams that report one percent-agreement figure won't.

Resources

Agreement statistics: primary sources

• A Coefficient of Agreement for Nominal Scales – Cohen (1960), Educational and Psychological Measurement
  • https://doi.org/10.1177/001316446002000104
• Measuring Nominal Scale Agreement Among Many Raters – Fleiss (1971), Psychological Bulletin
  • https://doi.org/10.1037/h0031619
• Computing Krippendorff's Alpha-Reliability – Krippendorff (2011), University of Pennsylvania
  • https://www.asc.upenn.edu/sites/default/files/2021-03/Computing%20Krippendorff%27s%20Alpha-Reliability.pdf
• The Measurement of Observer Agreement for Categorical Data – Landis & Koch (1977), Biometrics
  • https://doi.org/10.2307/2529310

Thresholds and their critiques

• Interrater Reliability: The Kappa Statistic – McHugh (2012), Biochemia Medica
  • https://pmc.ncbi.nlm.nih.gov/articles/PMC3900052/
• Inter-Coder Agreement for Computational Linguistics – Artstein & Poesio (2008), Computational Linguistics
  • https://aclanthology.org/J08-4004/
• High Agreement but Low Kappa: I. The Problems of Two Paradoxes – Feinstein & Cicchetti (1990), Journal of Clinical Epidemiology
  • https://doi.org/10.1016/0895-4356(90)90158-L
• High Agreement but Low Kappa: II. Resolving the Paradoxes – Cicchetti & Feinstein (1990), Journal of Clinical Epidemiology
  • https://doi.org/10.1016/0895-4356(90)90159-M

Agreement in LLM evaluation

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena – Zheng et al. (2023), NeurIPS Datasets and Benchmarks
  • https://arxiv.org/abs/2306.05685
• Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges – Thakur et al. (2025), GEM² Workshop at ACL
  • https://arxiv.org/abs/2406.12624
• Validating LLM-as-a-Judge Systems under Rating Indeterminacy – Guerdan et al. (2025), NeurIPS
  • https://arxiv.org/abs/2503.05965
• Open Benchmark of AI Impact on Humans (ImpactBench) – Archiwaranguprok et al. (2026), arXiv preprint
  • https://arxiv.org/abs/2610.20690

Gold answers and disagreement

• Cheap and Fast, but Is It Good? Evaluating Non-Expert Annotations for Natural Language Tasks – Snow et al. (2008), EMNLP
  • https://aclanthology.org/D08-1027/
• Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation – Aroyo & Welty (2015), AI Magazine
  • https://doi.org/10.1609/aimag.v36i1.2564
• The "Problem" of Human Label Variation – Plank (2022), EMNLP
  • https://aclanthology.org/2022.emnlp-main.731/

Kili

• ImpactBench Data Story – what an expert audit of an LLM judge shows
  • https://kili-technology.com/blog/data-story-impactbench-llm-judge-expert-audit
• Who Judges the Judge? – raw agreement vs chance-corrected agreement for LLM judges
  • https://kili-technology.com/blog/data-story-who-judges-the-judge-llm-as-a-judge
• How to Build Golden Datasets for Testing, Fine-tuning, and Evaluating AI Models
  • https://kili-technology.com/blog/how-to-build-golden-datasets-for-testing-fine-tuning-and-evaluating-ai-models
• Top Data Quality Metrics for Assessing Your Labeled Data
  • https://kili-technology.com/blog/top-data-quality-metrics-for-assessing-your-data-labeling-quality
• DPO vs RLHF Explained – agreement thresholds for preference labels
  • https://kili-technology.com/blog/dpo-vs-rlhf
• How to Build an SFT Dataset with Domain Experts – review rubrics and acceptance rules for written answers
  • https://kili-technology.com/blog/supervised-fine-tuning-dataset
• Kili docs: Consensus overview
  • https://docs.kili-technology.com/docs/consensus-overview
• Kili docs: Honeypot overview
  • https://docs.kili-technology.com/docs/honeypot-overview
• Kili docs: Calculation rules for quality metrics
  • https://docs.kili-technology.com/docs/calculation-rules-for-quality-metrics
• Kili docs: Best practices for quality workflows
  • https://docs.kili-technology.com/docs/quality-management-best-practices-for-data-labeling
• Kili docs: Analytics page
  • https://docs.kili-technology.com/docs/analytics-page
• Kili docs: Exporting project data
  • https://docs.kili-technology.com/docs/exporting-project-data
• Kili docs: Labeling audio assets
  • https://docs.kili-technology.com/docs/labeling-audio-assets
• Kili changelog, 7 October 2026 (v26.2.12) – parallel blind consensus
  • https://docs.kili-technology.com/changelog/october-7th-2026
• Kili changelog, 21 October 2026 (v26.2.13) – add or remove assets from consensus on the Queue page
  • https://docs.kili-technology.com/changelog/october-21st-2026
• Talk to the Kili team
  • https://kili-technology.com/get-started/evaluation

Frequently Asked Questions

What is a good inter-rater reliability score?

For a chance-corrected statistic such as Cohen's kappa or Krippendorff's alpha, 0.8 or above is the common target for labels you will rely on. Values from 0.67 to 0.8 support tentative conclusions. The often-quoted Landis and Koch scale calls 0.61 to 0.80 "substantial", but critics such as McHugh consider anything under 0.60 inadequate for health research.

What is the difference between Cohen's kappa and Fleiss' kappa?

Cohen's kappa measures agreement between exactly two raters who rate every item. Fleiss' kappa handles three or more raters, as long as each item gets the same number of ratings; the people can differ from item to item. For missing ratings or ordinal scales, use Krippendorff's alpha.

Why not just use percent agreement?

Because raters agree some of the time by chance, and the chance rate rises when one label dominates. Two raters can agree 90% of the time and still score a kappa below zero if almost every item gets the same label. Report percent agreement alongside a chance-corrected figure.

Can raters agree and still be wrong?

Yes. Agreement shows consistency, not correctness: raters can share the same misreading of a guideline. That is why you also need gold answers (honeypots) that compare each rater with a trusted reference.

Do consensus and honeypot work on every data type in Kili?

They score classification, transcription, named entity, OCR (optical character recognition) and object detection jobs (boxes, polygons and segmentation masks). Points, lines and entity relations aren't scored yet, and neither feature is available on audio projects, which are in beta.