.png)
_logo%201.svg)
AI Summary
- Average scores hide regressions; count the items the new model broke, not just the net gain.
- Score both versions on the same items, with the same pinned judge, and report the paired difference with a confidence interval.
- A 3-point gain on 500 items can still be noise.
- Experts read every regression before release.
Introduction
In April 2025, OpenAI shipped a GPT-4o update to ChatGPT and rolled it back days later because the model had become sycophantic. Its postmortem explains why the release went out anyway. The offline evaluations, "especially those testing behavior", looked good. A small A/B test suggested users liked the new model. Some of the expert testers who tried it before launch said the behaviour "felt" slightly off. The metrics won.
That is the hard part of LLM regression testing. After any model update, the question sounds simple: did it help? The update might be a new checkpoint, a new post-training dataset, a new system prompt or a new provider snapshot. Answering the question needs three things most teams don't have at once. You need a fixed set of test items graded by people who know the domain. You need a comparison that holds everything constant except the change. And you need a rule for when a difference is large enough to believe.
This guide covers each of them, then shows how to run the expert grading in Kili.
What Is LLM Regression Testing?
LLM regression testing re-runs a fixed set of evaluation items on every new model version and checks, item by item, that nothing that used to work has stopped working. The name comes from software testing, where a regression suite guards old behaviour while new features ship.
Machine learning research has a name for the failure it targets: a negative flip, an example the old model got right and the new model gets wrong. Positive-Congruent Training is a CVPR 2021 paper on image classifiers. In it, Yan et al. argue that reducing these flips "can be as important in practice as reducing its overall error". A new model can be more accurate on average and still break cases your users relied on.
The same thing happens with language models. Apple researchers working on MUSCLE updated a base model, for example from Llama 1 to Llama 2. Task adapters built on it then showed negative flips: "previously correct instances are now predicted incorrectly". A model can also change without any action on your side. A Stanford and Berkeley study, How Is ChatGPT's Behavior Changing over Time?, tested GPT-4 on identifying prime numbers. Accuracy fell from 84% in the March 2023 version to 51% in the June 2023 version.
How Do You Catch Regressions After a Model Update?
You catch regressions with a regression suite. It is a fixed, versioned set of items with expert-approved answers or grading rubrics. You run it on both the old and the new version and compare the results item by item. Four parts make it work.
1. A fixed holdout that never trains the model
A holdout is a test set kept out of training. The suite is only useful if the model has never been tuned against it, so keep it out of every training set and every prompt-engineering loop. Our guide to golden datasets covers how to source and version these items. Our custom AI benchmark playbook covers how many you need and how to grade them. Include the tasks users send, edge cases that have failed before, and behaviours you rarely measure, such as refusals, tone and agreeing with a wrong premise.
2. A per-item comparison, not just a score
For each item, record whether the old version passed and whether the new version passed. Then count four groups:
- both pass
- both fail
- fixed: the old version failed and the new one passes
- broken: the old version passed and the new one fails
The change in the headline score is fixed minus broken, divided by the number of items. Two updates with the same net gain can have very different broken counts. The broken list is your regression report.
3. Slices and behaviour checks
Break results down by task type, customer segment, language and difficulty. A 2-point overall gain can hide a 10-point loss on one slice. Add behaviour evals for the failure modes that metrics miss. After its rollback, OpenAI added sycophancy evaluations to its deployment process. It also wrote that "personality and other behavioral issues should be launch blocking".
4. Monitoring after release
In September 2025, Anthropic published a postmortem of three infrastructure bugs that degraded some Claude responses for several weeks. At the worst hour, one of them affected 16% of Sonnet 4 requests. Anthropic's evaluations didn't capture what users reported, partly because the model often recovers well from isolated mistakes. Its fixes included continuous quality evaluations on production systems. Run a slice of your suite against the live system on a schedule, not only against the release candidate.
How Do You Compare Two Model Versions Fairly?
A fair comparison changes one thing and holds everything else constant. In practice:
- Same items: both versions answer the identical holdout. Never compare a score from last quarter's subset with a score from this quarter's.
- Same settings: the same system prompt, temperature, maximum length, tools and retrieval index. For a RAG (retrieval-augmented generation) system, score retrieval and generation separately so you know which one moved. Our RAG evaluation guide shows how.
- Same grader, in the same run: re-grade the old version's outputs alongside the new ones. A score stored three months ago came from a judge prompt, judge model and grading team that may have changed since.
- Blind and order-balanced: strip version names before grading. In pairwise comparisons, show each pair in both orders.
- Paired analysis: compare the two versions item by item and report the mean difference with a confidence interval (next section).
Pairwise comparison suits this well. Graders, human or model, find "which answer is better?" easier to judge than an absolute score. Public leaderboards use the same format to answer a different question.
What public leaderboards measure instead
Chatbot Arena ranks models from crowdsourced pairwise votes. In December 2023, its maintainers switched from online Elo to a Bradley-Terry model. Elo is the chess rating method, updated after every game. Bradley-Terry is a statistical model that fits all win/loss records at once, so "game order does not matter". The maintainers also bootstrap the confidence intervals, so a model with fewer votes gets a wider interval.
The methods are sound. The caveats concern what gets measured. An August 2024 style-control analysis found that answer length and markdown formatting move the rankings. Once both were controlled, Grok-2-mini fell from 6th to 18th. The Leaderboard Illusion reported that some providers privately test many variants before release, including 27 private variants from Meta before Llama 4. Meta's public Arena entry was an experimental chat version, "optimized for conversationality". TechCrunch reported that once the released model was scored, it ranked below GPT-4o, Claude 3.5 Sonnet and Gemini 1.5 Pro.
A public rank tells you how a crowd felt about a variant on the crowd's prompts, not whether your update helped your users. For that, run the same pairwise method on your own holdout, with graders who know the domain.
When Is a Difference Between Two Model Versions Real?
A difference is real when its confidence interval excludes zero and someone has read the regressions behind it. A confidence interval is the range the true difference probably falls in. Anthropic researcher Evan Miller sets out the statistics in Adding Error Bars to Evals. His starting point is that "evaluations are experiments", so each score is an estimate with uncertainty.
Use paired differences
When both versions answer the same items, compute the difference item by item, then take its mean and standard error. Miller recommends "using the paired version of the standard error estimate wherever practicable". Pairing removes the variance that comes from some questions being harder than others. Anthropic's summary of the paper notes that frontier models' per-question scores often correlate between 0.3 and 0.7. So the reduction is substantial, and it costs nothing.
Here is a worked example with illustrative numbers. On a 500-item pass/fail holdout, the new version fixes 40 items and breaks 25. The net gain is 15 items, or 3 points. The paired 95% confidence interval runs from about −0.2 to +6.2 points. It includes zero, so the evidence for an improvement is weak, and 25 items that used to pass now fail. With the same rates on 1,000 items, the interval would be about +0.8 to +5.2 points. That's consistent with Miller's power analysis. It needs about 969 questions to detect a 3-point difference at 80% power (an 80% chance of detecting a real difference that size). Miller suggests new evals have "at least 1,000 questions".
Account for everything that makes the interval wider
The small-holdout row matters for expert-graded sets, which often hold a few hundred items. A 2025 ICML position paper makes the point in its title: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints. The CLT (central limit theorem) is the basis of standard intervals. On small evals, the authors find those intervals end up "usually dramatically underestimating uncertainty". The problem predates LLMs. With Little Power Comes Great Responsibility (EMNLP 2020) found that "underpowered experiments are common in the NLP literature", and that typical human rating studies are too small to detect modest model differences.
How Do You Keep an LLM Judge From Drifting?
Most teams grade part of the suite with an LLM judge. The judge is part of the measuring instrument, and it has known biases.
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (NeurIPS 2023) documented three:
- Position bias. When two very similar answers swapped places, GPT-4 gave the same verdict in only 65% of cases.
- Verbosity bias. The authors padded answers with a rephrased copy of their own list. GPT-3.5 and Claude-v1 preferred the padded version 91.3% of the time; GPT-4 only 8.7%.
- Self-enhancement bias: a judge favouring answers from its own model.
The ACL 2024 paper Large Language Models Are Not Fair Evaluators used ChatGPT as the judge. Just by changing the answer order, Vicuna-13B could beat ChatGPT on 66 of 80 queries. Panickssery et al. (NeurIPS 2024) found that judges recognise their own outputs, and the better they recognise them, the more they favour them.
These biases matter most when the update changes what the judge is biased towards. A post-training run that makes answers longer will win more verbosity-biased comparisons without being better. That is why Length-controlled AlpacaEval exists: the original AlpacaEval tended to "favor models that generate longer outputs".
Controls that work:
- Pin the judge to a dated model snapshot and a versioned prompt. If the judge is an API model, a provider update can change how it grades, as the GPT-4 drift study showed.
- Judge every pair in both orders, and count a win only when it holds in both. MT-Bench's authors call a split result a tie.
- Don't let a model judge its own family's outputs when you can avoid it.
- Report answer length next to every win rate.
- Re-check the judge against expert grades on a calibration sample each release. Our data story Who Judges the Judge? shows how a judge can be consistent and still disagree with the experts it stands in for.
Where Do Expert Spot Checks Fit?
Statistics tell you whether a difference is larger than noise. They don't tell you whether the 25 newly failing items matter, or whether the judge scored them correctly. That's the expert's job. It's also where OpenAI's GPT-4o release went wrong: the expert testers' unease was the only signal pointing the right way, and the team overruled it.
Put expert review at fixed points in the release process, not as an optional sanity check:
- Every regression: experts read each item the new version broke. Some will be grading errors, some acceptable trade-offs, and some will block the release.
- A sample of fixes: a gain the judge awards for a longer or more confident answer is not a gain.
- Judge disagreements: items where the judge's two orderings disagree, or where judge and expert disagree on the calibration sample.
- A small random sample, to catch problems no rule targets.
For OpenAI's HealthBench, 262 physicians wrote rubrics for 5,000 health conversations. OpenAI then checked its model grader against physician grading. Agreement between the model and physicians, and among physicians, both ranged from 55% to 75%. Our AI model evaluation guide covers when a task needs a domain expert grader at all.
How Do You Tell if New Post-Training Data Improved the Model?
Train an ablation: two runs that differ only in the new data. Use the same base model, recipe and hyperparameters, once with the new data and once without. Then compare the two checkpoints on a holdout the data never touched, using the paired method above. If anything else changes between the two runs, you can't tell what caused the difference.
Decide on one set, confirm on another
Ai2's Tulu 3 post-training project used two evaluation suites. A development suite guided choices about data mixtures, algorithms and hyperparameters. A separate unseen suite checked for overfitting. The authors write that "we did not examine scores on our unseen set" during development. Every time you look at a holdout to pick a data mix, it becomes a little more like a training set. An expert-graded unseen set, opened only for the final decision, is the one you can believe.
Measure the noise before reading the gain
Training the same recipe with a different random seed changes scores. A Meta-led study, Quantifying Variance in Evaluation Benchmarks, notes that variance determines "whether differences in performance are meaningful". Yet benchmarks are rarely checked for it. Ai2's Signal and Noise study covered 375 models. Benchmarks with a better signal-to-noise ratio supported more reliable decisions from small experiments, and removing noisy subtasks helped. If a data change moves the score less than two seeds of the same recipe disagree, you haven't shown anything.
Check the target slice and everything else
New data usually targets a gap: a domain, a format, a failure mode. Check that slice for the gain, then check every other slice for negative flips. New SFT (supervised fine-tuning) examples that teach a longer answer format can lift the target task and lengthen answers everywhere. OpenAI's sycophancy postmortem describes the same pattern at a larger scale. Several training changes each looked beneficial on their own, including a new reward signal from users' thumbs-up and thumbs-down data. Together, they tipped the model into sycophancy. Our guides to building an SFT dataset with domain experts and to DPO vs RLHF cover how the data itself shapes those side effects.
How Do You Run This Workflow in Kili?
Kili doesn't train or serve your models. It is where domain experts grade their outputs.
To grade a release candidate against the current version, import each holdout item as one text asset with both versions' answers. Remove version names and randomise the answer order first. Experts grade with classification jobs that follow your rubric: pass/fail, a preference between the two answers, an error category. A transcription job captures a short justification. A multi-step workflow adds one or more review steps after grading. For each review step, you choose where an asset goes if it doesn't pass review. Enforce Step Separation, on by default, stops a member from working on more than one step of the same asset. No one reviews their own grade.
Two quality checks apply to the graders themselves. Consensus sends a set percentage of assets to several experts, who grade blind, and measures their agreement. That tells you whether the rubric is clear enough to compare versions with. Honeypot, enabled on one workflow step, calibrates each grader against reference labels you upload through the SDK. Kili stores annotations and reviews as separate label types, each with its author and date. A release decision can then be traced back to the grades and graders behind it. Export the grades, join them to the item IDs, and compute the paired differences and intervals in your own stack.
Kili also has a dedicated LLM interface built for pairwise comparison. In it, experts rank and evaluate model responses in multi-turn conversations. The conversations are either generated live from two or more configured models or imported from production. This interface is in private beta: it isn't enabled by default, and access is on request.
Conclusion: The Holdout Is the Asset
Model versions come and go. The expert-graded holdout stays: the one measurement that means the same thing for the old model, the new model and the one after that. It is fixed, so comparisons are paired. It is graded by people who know the domain, so a "pass" means what users would mean. Store each grade with who gave it, which rubric version it used and who reviewed it. Then a release decision can be traced back to the items that drove it.
Teams that treat evaluation as a dashboard ask whether the number went up. Teams that treat the holdout as an asset ask which items changed, whether the change is bigger than the noise, and whether an expert agrees. The second group hears about regressions from its experts before it hears about them from users. OpenAI's postmortem shows the gap the first group leaves: the metrics were all there, and no rule said when expert judgment overrides them.
Resources
Statistics for model comparison
- Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations – Miller (2024), arXiv preprint, Anthropic
- A statistical approach to model evaluations – Anthropic (2024)
- Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints – Bowyer, Aitchison & Ivanova (2025), ICML
- With Little Power Comes Great Responsibility – Card et al. (2020), EMNLP
- Quantifying Variance in Evaluation Benchmarks – Madaan et al. (2024), arXiv preprint, Meta
- Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation – Heineman et al. (2025), arXiv preprint, Ai2
Regressions and model updates
- Positive-Congruent Training: Towards Regression-Free Model Updates – Yan et al. (2021), CVPR
- MUSCLE: A Model Update Strategy for Compatible LLM Evolution – Echterhoff et al. (2024), arXiv preprint, Apple
- How Is ChatGPT's Behavior Changing over Time? – Chen, Zaharia & Zou (2023), arXiv preprint
- Expanding on what we missed with sycophancy – OpenAI (2025), vendor postmortem
- A postmortem of three recent issues – Anthropic (2025), vendor postmortem
LLM judges and their biases
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena – Zheng et al. (2023), NeurIPS Datasets and Benchmarks
- Large Language Models are not Fair Evaluators – Wang et al. (2024), ACL
- LLM Evaluators Recognize and Favor Their Own Generations – Panickssery, Bowman & Feng (2024), NeurIPS
- Length-Controlled AlpacaEval – Dubois et al. (2024), COLM
Public leaderboards
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference – Chiang et al. (2024), arXiv preprint
- Chatbot Arena: New models & Elo system update – LMSYS (2023)
- Does Style Matter? Disentangling style and substance in Chatbot Arena – Li, Angelopoulos & Chiang (2024), LMSYS
- The Leaderboard Illusion – Singh et al. (2025), arXiv preprint
- Meta's vanilla Maverick AI model ranks below rivals on a popular chat benchmark – Wiggers (2025), TechCrunch
Holdouts and expert grading
- Tulu 3: Pushing Frontiers in Open Language Model Post-Training – Lambert et al. (2024), arXiv preprint, Ai2
- HealthBench: Evaluating Large Language Models Towards Improved Human Health – Arora et al. (2025), arXiv preprint, OpenAI
Kili
- How to Build Golden Datasets for Testing, Fine-tuning, and Evaluating AI Models
- How to Build a Custom AI Benchmark: A 5-Phase Playbook
- RAG Evaluation Guide: Measuring Retrieval and Generation as Separate Problems
- Who Judges the Judge? A Data Story on LLM-as-a-Judge
- AI Model Evaluation Guide: Methods, Metrics, and Why It Determines Production Success
- How to Build an SFT Dataset with Domain Experts
- DPO vs RLHF Explained
- Workflow configuration – Kili documentation
- Consensus overview – Kili documentation
- Honeypot overview – Kili documentation
- Labeling LLM data (private beta) – Kili documentation
- Talk to the Kili team
Frequently Asked Questions
What is LLM regression testing?
LLM regression testing re-runs a fixed set of evaluation items on each new model version. It compares the results item by item with the previous version. It looks for changes in the overall score and for negative flips: items the old version handled correctly and the new one gets wrong.
How many test items do you need to compare two LLM versions?
It depends on the smallest difference you want to detect. In Evan Miller's power analysis, detecting a 3-point difference with 80% power took about 969 questions. He suggests new evals have at least 1,000. Expert-graded holdouts are often smaller, so only larger differences can be confirmed. Below a few hundred items, standard confidence intervals also become too narrow.
Can you use an LLM as a judge for regression testing?
Yes, with controls. Pin the judge model and prompt, and judge pairwise comparisons in both orders. Report answer length next to win rates. Check the judge against expert grades on a calibration sample each release, and have experts read every item the new version broke.
What is a negative flip?
A negative flip is an item the previous model version got right and the new version gets wrong. A new model can score higher overall and still have many negative flips, which is why regression suites compare versions item by item.
Should you trust Chatbot Arena rankings when choosing a model update?
Use them as a signal of crowd preference on general prompts, not as a test of your update. Arena rankings are sensitive to answer length and formatting, and providers can test private variants before release. Compare versions on your own holdout, graded by people who know your domain.
Ready to Put Experts on Your Regression Suite?
Kili lets domain experts grade model outputs in multi-step review workflows, with consensus and honeypot checks on the graders themselves. Talk to the Kili team about setting up an expert-graded holdout for your next model update.
