.png)
_logo%201.svg)
AI Summary
- Exact-match agreement overstated 21 LLM judges by 33–41 points versus Cohen's kappa on MT-Bench.
- Two production judges scored ≥0.95 test–retest while showing strong position bias.
- Expert and crowd pools split on 23.6% of items both called unanimously.
- Six-model leaderboards hid that split; twenty-model ones almost never would.
Introduction
Most teams validate an LLM judge with one number: how often it agrees with human labels. If that figure clears 80%, the judge goes into production and starts scoring model outputs, fine-tuning candidates and prompt variants at a scale no human team could match.
Two preprints published this summer suggest that number hides more than it shows. In June, Justin Norman, Michael Rivera and Alex Hughes ran 21 LLM judges through roughly 541,000 individual judgments on three benchmarks. They found that raw agreement overstated judge quality by 33 to 41 percentage points on MT-Bench once chance was accounted for, and that some judges were almost perfectly repeatable while being reliably biased. They called the paper Reliability without Validity.
In August, Anik Jha went one step earlier. When a benchmark says a judge "agrees with humans," which humans? His annotator-pool audit found that expert and crowd annotators disagree on nearly a quarter of the items each group felt unanimous about, and that LLM judges drift toward the crowd.
Together, the two studies describe a judge that can be consistent and wrong, scored against a gold standard that depends on who made it. The checks most teams run catch neither problem, and both can be measured before an LLM judge is trusted with decisions that matter.
What Did 541,000 LLM-Judge Decisions Actually Measure?
Norman, Rivera and Hughes tested 21 judge models from OpenAI, Google, Anthropic, Meta, Alibaba, Mistral, DeepSeek and several other providers. The cohort ran from small open-weight models like Qwen 3 8B to frontier systems like Claude Opus 4.6, GPT-5.4 and Gemini 3.1 Pro. Each judge was put through three protocols on MT-Bench, JudgeBench and RewardBench across 118 runs:
- Agreement: does the judge pick the same winner as the human reference label?
- Consistency: does it give the same verdict when asked again?
- Bias audit: does its verdict change when you swap the order of the two answers, or make one longer?
Most teams run some version of the first protocol. Fewer run the second. Almost nobody runs the third in production. Running all three on the same judges shows where the protocols disagree.
On MT-Bench, the reference labels are 2,391 pairwise comparisons with expert human judgments. A judge that scores poorly there is falling short of experts specifically.
Why Does Raw Agreement Overstate LLM-as-a-Judge Quality?
Because a judge can match human labels by accident, and percent agreement counts those accidents as skill.
Cohen's kappa corrects for this. It asks how much better than chance the judge agrees, given how often each label appears. A judge that always picks "tie" on a dataset where most items are ties will post a high match rate and a kappa near zero.
The 2023 MT-Bench paper by Lianmin Zheng and colleagues set the reference point most teams still use: GPT-4 as judge reached "over 80% agreement, the same level of agreement between humans." That figure is raw agreement. The 2026 study recomputes the same kind of number with chance correction, and the gap is large.
Gemini 3.1 Pro, the judge with the smallest gap on MT-Bench, matched the expert label 84.9% of the time. Its kappa was 0.511. GPT-4o-mini matched 80.9% of the time with a kappa of 0.396.
The authors call the effect kappa deflation, and they found it on every judge and every benchmark. The size varies: 33 to 41 points on MT-Bench, 8 to 39 on JudgeBench, 6 to 21 on RewardBench. So a judge's apparent quality depends on the label distribution of whatever you test it on.
Earlier work saw the same effect. A 2024 study, Judging the Judges by Aman Singh Thakur and colleagues, observed that "judges with high percent agreement can still assign vastly different scores" and called for metrics beyond percent alignment. The 2026 study adds scale: 21 judges and three benchmarks, with the gap present on every one.
Can an LLM Judge Be Consistent and Still Wrong?
Yes, and the study names two that were.
Qwen 3 8B and Gemini 2.5 Flash, both described as production-deployed, gave the same verdict on repeated runs at least 95% of the time. By the consistency check most teams rely on, they were excellent. On MT-Bench, they also showed position-bias scores of 0.192 and 0.125. The score measures how far the judge's preference for the first-shown answer drifts from an even 50/50 split.
The authors describe this as "highly reproducible but not valid." The plainer version: the judge is consistent and wrong. Re-running it gives you the same biased answer with high confidence.
Test–retest reliability is also the cheapest check to run: it needs no human labels and no counterbalancing. Teams that stop there will select for judges that are stable, which is a different property from judges that are right.
Rankings were no more stable. The abstract reports judges moving up to 14 positions across benchmarks. In the body of the paper, Llama 3.3 70B goes from 5th on MT-Bench to 20th on JudgeBench. A judge you chose on one benchmark may be a poor pick for your task.
Swapping in a newer model doesn't reliably fix this either. A July preprint by Zongyou Yang, Yinghan Hou and Xiaokun Yang, When the Judge Changes, So Does the Measurement, found that stronger judges reduce position and verbosity bias without removing it, and that "judge upgrades are not interchangeable." Every judge change is a measurement change and needs revalidating.
Whose Labels Count as Gold?
Every agreement number above assumes the human labels are correct. Jha's audit tests that assumption directly.
He used MultiPref, a preference dataset in which each comparison was labelled twice by crowdworkers and twice by screened domain experts. He then isolated 2,885 items where both pools were internally unanimous: the two experts agreed with each other, and so did the two crowdworkers. These are the items any team would treat as settled.
On 23.6% of them, the expert majority and the crowd majority gave different labels. On 9.2%, they named opposite winners. MT-Bench told a similar story: the benchmark's own authors and recruited expert annotators differed on 30.5% of comparable cells and reversed each other on 8.5%.
So "agreement with human labels" is underspecified until you say which humans. A judge that scores well against crowd labels may be scoring well against the wrong reference for an expert task.
Jha then checked which pool LLM judges resemble. Three open-weight judges (two Qwen3.6 models and one Gemma-4 model, from two vendors) all agreed more with the crowd majority than the expert majority, by 3.7 to 6.9 points on self-consistent items. The judges tested were mid-size open models, not frontier APIs, so the result shouldn't be stretched further than that. But the direction held for all three: the judges sat closer to the cheaper pool.
Earlier work in expert domains points the same way. When Annalisa Szymanski and colleagues asked dietitians and mental-health professionals to compare their preferences against an LLM judge, the experts agreed with it only 68% and 64% of the time. The authors stressed "the importance of keeping human experts in the evaluation process."
Jha's first recommendation follows directly: any judge-agreement figure should name the annotator pool that defined the reference labels. It costs one line in a methods section.
Why Do Small Leaderboards Hide Annotator Disagreement?
Despite the item-level disagreement, Jha found that six-model leaderboards came out identical whichever pool you used (Kendall tau of 1.00, no model displaced). On the surface, annotator identity doesn't matter.
He then showed why that reassurance doesn't travel. Switching pools moved a model's win rate with a standard deviation of about 1.9 points. One adjacent pair on the leaderboard sat only 0.8 points apart and had a 38% chance of swapping. Under the same perturbation, the probability of displacing at least one model rose to 0.86 with ten models and 0.9997 with twenty.
A six-model comparison is stable mostly because there are few close pairs to flip. Many teams end up comparing more than that once prompt versions, fine-tuning checkpoints, retrieval settings and vendor models are all in the running. At twenty candidates, disagreement between annotator pools will almost certainly reorder the results.
Jha's recommendation: "Do not infer per-item reliability from leaderboard stability."
Agentic evaluation shows the same pattern from another angle. Benchmarking the Benchmarks, a June preprint from Vishvesh Bhat and colleagues, audited 496 expert-reviewed tool-calling tasks and found the automated evaluator disagreed with human experts on 92 of them, or 18.5%. Re-running the same LiveMCPBench setup 23 times produced scores from 57.9% to 76.8%, an 18.9-point spread. When run-to-run noise is that wide, a few points of difference between two agents tells you very little.
If Experts Disagree Too, What Should Teams Measure?
Swapping crowd labels for expert labels doesn't settle the problem either.
In a January 2026 study, Kiana Jafari and colleagues had three certified psychiatrists rate LLM responses using a calibrated rubric. Their inter-rater reliability (ICC) ranged from 0.087 to 0.295, well below what high-stakes assessment normally requires. On one factor, Krippendorff's alpha was −0.203: the psychiatrists disagreed more than chance would predict. The disagreement was systematic, reflecting what the authors call "coherent but incompatible individual clinical frameworks."
Experts can also be unsure what they're measuring until they start. Shreya Shankar and colleagues documented criteria drift: people need criteria to grade outputs, but grading outputs is how they discover their criteria. A rubric written before anyone looks at the data will change once someone does.
Hiring experts is necessary, then, and it isn't sufficient. Expert judgment needs the same measurement as the LLM judge it is calibrating: agreement between experts, corrected for chance, tracked over time, with disagreements surfaced rather than averaged away.
The practitioner literature lands in the same place. The Hugging Face LLM-judge cookbook by Aymeric Roucher measures the agreement between two human raters before tuning the judge, and finds it at only 0.563. The judge eventually reached 0.843 against those labels. Without the human baseline, you'd have no way to know whether the judge's errors were its own or inherited. Hamel Husain and Shreya Shankar's evals FAQ recommends a single domain expert as the final authority on quality, and validating judges by true-positive and true-negative rate rather than overall agreement.
How Should You Validate an LLM Judge Before Trusting It?
Taken together, the studies point to a short protocol.
Start with expert labels, and name the pool
Build a calibration set labelled by the people who actually hold the domain expertise: the clinician, the underwriter, the quality engineer. Record who labelled each item. Keep per-annotator labels rather than collapsing to a majority vote, so you can stratify later.
Measure the experts before you measure the judge
Compute inter-annotator agreement with a chance-corrected metric (Cohen's kappa for two raters, Krippendorff's alpha or Fleiss' kappa for more). If experts don't agree, the judge has nothing stable to learn from. Fix the rubric and the guidelines first; prompt tuning comes after.
Report kappa next to raw agreement
Our own guide to building a custom AI benchmark suggests checking a judge against experts on a 50–100 item golden set and looking for at least 75% agreement. That threshold is a reasonable floor, but on the evidence above it should always be read alongside a chance-corrected figure. A 75% match on a skewed label set can be close to chance.
Counterbalance order and probe for bias
Run every pairwise comparison in both orders. If a judge's verdict flips when the answers swap places, it is responding to position, and its scores on that item can't be trusted.
Revalidate on every change
A new judge model, a new rubric version or a new data distribution all count. Treat each one as a new instrument.
Keep humans in the loop on the disagreements
The items where judge and expert diverge are the most informative data you have. Route them to expert review, and feed the resolved labels back into the calibration set. Our report on building trusted GenAI with LLM-as-a-judge and human-in-the-loop workflows sets out this division of labour: the judge triages, and people make the calls it can't.
Most of this protocol concerns the human side of the comparison, which is where an annotation platform does its work. In Kili Technology, consensus measures agreement between multiple annotators working on the same asset, and consensus labeling runs blind, so annotators can't anchor on each other. Honeypots score each annotator's labels against a trusted ground-truth reference. Multi-step review enforces step separation by default, so no one reviews their own work. Before you trust a judge, you need to trust the labels you're judging it against.
Reliability Is Cheap. Validity Needs Experts and Measurement.
The two studies point to a gap between what is easy to check and what matters. Test–retest reliability needs no humans. Raw agreement needs some humans, any humans. Validity needs the right humans, measured properly, and that's the expensive part teams are tempted to skip.
As LLM judges move from leaderboard tooling into reward models and production monitoring, that gap gets more expensive. A biased judge in a post-training loop mis-scores more than a benchmark. Its preferences become the signal the model learns from. The teams that avoid this will be the ones that can answer three questions for any judge they deploy: who defined correct, how much those people agreed, and how far the judge sits from them once chance is removed.
The expert makes the judgment. The platform has to measure whether that judgment was sound.
Resources
The Two Spine Studies
- Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias – Norman, Rivera & Hughes (2026), arXiv preprint
- Whose Gold? Annotator-Pool Disagreement Is Large at the Item Level, and Hidden by Small Leaderboards – Jha (2026), arXiv preprint
LLM-as-a-Judge Reliability Research
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena – Zheng et al. (2023), NeurIPS Datasets & Benchmarks
- Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges – Thakur et al. (2024), arXiv preprint
- When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability – Yang, Hou & Yang (2026), arXiv preprint
- Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation – Bhat et al. (2026), arXiv preprint
Expert Judgment and Annotator Disagreement
- Expert Evaluation and the Limits of Human Feedback in Mental Health AI Safety Testing – Jafari et al. (2026), arXiv preprint
- Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks – Szymanski et al. (2024), arXiv preprint
- Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences – Shankar et al. (2024), UIST 2024
Practitioner Guidance
- Using LLM-as-a-judge for an automated and versatile evaluation – Aymeric Roucher, Hugging Face Open-Source AI Cookbook
- LLM Evals FAQ – Hamel Husain & Shreya Shankar
Kili Technology
- How to Build a Custom AI Benchmark Guide: A 5-Phase Playbook – Kili Technology (May 2026)
- Building Trusted GenAI with LLM-as-a-Judge and Human-in-the-Loop Workflows – Kili Technology report
- Contact the Kili team
Ready to Build an Evaluation You Can Defend?
If you're designing a custom benchmark or calibrating an LLM judge against domain experts, start with our five-phase playbook for building a custom AI benchmark. It covers golden-set design, expert rubrics and per-criterion agreement tracking. To set up expert-in-the-loop evaluation on your own data, talk to the Kili team.
