AI Model Evaluation Workflow Guide: From Build to Monitor

Introduction

A team ships a detector. The labeling project that produced its training data goes quiet: annotators reassigned, ontology frozen, project archived. A quarter later someone asks whether field accuracy has moved, and the best answer available is a count of complaints from the site with the oldest cameras.

Most teams already own the machinery for a production AI model evaluation workflow and call it something else. The build phase runs collect data, define an ontology, get experts to label it, measure agreement, review, ship. The monitor phase runs sample production output, score it against a rubric, get experts to correct it, measure drift, feed corrections back. Same ontology work, same reviewer roles, same agreement statistics. The input changed from raw imagery to model predictions with imagery attached. The machinery did not.

Computer vision is where the argument is easiest to settle, because the field has spent fifteen years measuring what happens when a model meets a new site. A systematic review of 86 external-validation studies of deep learning for radiologic diagnosis found 81% reported some performance drop outside their development data, 49% a drop of at least 0.05, and 24% more than 0.10. Since these are published studies and publication bias runs toward flattering results, that is the optimistic version.

The hard part is staffing the loop. Both obvious ways to do it fail measurably: automated screening cannot certify itself, and a reviewer shown the model's answer first does worse than one shown nothing. This guide covers both failures and the evaluation framework that routes around them.

Why Did the Benchmark Your Detector Shipped On Stop Being Evidence?

Because the conditions it measured were narrower than production.

The canonical demonstration is Zech and colleagues' pneumonia-detection study across 158,323 chest radiographs. A model scoring AUC 0.931 internally dropped to 0.815 at a third hospital it had never seen. Performance degraded in three of five natural comparisons, so read it as a strong tendency, short of a law. The other half of the finding is not contested: the network could identify which hospital system an image came from with near-perfect accuracy. It had learned the site, and prevalence correlated with the site.

Sensor and weather conditions do the same thing more bluntly. Benchmarking detectors against corrupted versions of Pascal VOC, COCO and Cityscapes, Michaelis and colleagues found that "standard object detection models suffer a severe performance loss on corrupted images", retaining 30 to 60% of original performance. The corruptions are synthetic data, so treat that range as a bound. The loss is still multiplicative.

Two results suggest the problem runs deeper than any particular shift. Recht and colleagues rebuilt the original ImageNet and CIFAR-10 collection protocols to make fresh test sets, and models still showed "accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet"; their reading is that the new images were marginally harder, not that everyone had overfit. Taori and colleagues then built a testbed of 204 models across 213 conditions and reported that "most current techniques provide no robustness to the natural distribution shifts in our testbed". Benchmark-measured robustness did not predict robustness in the wild.

A score recorded at launch is a historical fact about a distribution you no longer serve. ObjectNet randomized viewpoint, rotation and background across 50,000 images and cut detector performance 40 to 45%. Nothing about the objects changed. Only the photographer's habits did.

Is Your Ground Truth as Solid as Your Evaluation Metrics Assume?

Northcutt, Athalye and Mueller audited ten canonical test sets and found that "label errors comprise at least 6% of the ImageNet validation set", 2,916 of 50,000 examples. The errors reach the rankings: they show ResNet-18 overtaking ResNet-50 once the prevalence of originally-mislabeled test examples rises six percentage points. Model rankings are partly an artifact of annotation noise.

Every headline metric inherits the problem. Classification accuracy is the share of predictions that match the reference label, and F1-score is the harmonic mean of precision and recall against that same reference. When the reference is wrong, the evaluation scores are wrong with it.

Detection has its own version. Re-inspecting COCO-2017 masks, the COCO-ReM authors found imprecise boundaries, non-exhaustive instances and mislabeled masks, then re-evaluated fifty detectors against corrected annotations: "models that predict visually sharper masks score higher on COCO-ReM". The old annotations had been penalizing the better models.

Where the task is hard, "ground truth" is a spread. Joskowicz and colleagues had eleven radiologists produce 3,193 contours across 490 CT slices; across randomly chosen observer pairs, "the mean delineation volume overlap variability was 5–57%". That is volume overlap rather than Dice, at one institution, but the shape generalizes: for many perception tasks the reference has a distribution and your model's ceiling sits inside it.

So annotation agreement belongs in your error bars, not in a QA step upstream of them.

Can Automated Screening Run an AI Model Evaluation Workflow Alone?

Automated screening is the right first move. It is cheap, it runs on every frame, and automated scoring produces a number you can chart. Three findings limit what that number can be trusted to mean.

The first concerns using models as judges, the multimodal counterparts of the LLM judges now common in text evaluation. The MLLM-as-a-Judge benchmark found multimodal judges track human preference tolerably on pairwise A/B comparison while showing "significant divergence from human preferences in Scoring Evaluation and Batch Ranking". GPT-4V, the strongest judge tested, reached 0.490 similarity to human scoring against roughly 0.64 to 0.77 on pair comparison. Those absolute numbers date quickly; the task-dependence holds. Asking a model which of two outputs is better works far better than asking it to assign a score.

The second concerns feeding predictions back as labels. Arazo and colleagues named the mechanism: "naive pseudo-labeling overfits to incorrect pseudo-labels due to the so-called confirmation bias". Training on your own wrong predictions raises confidence in those errors and produces a model that resists correction. An auto-labeling loop with no independent human reference converges on its priors, not on truth.

The third undermines the fix most teams reach for. If confidence scores were reliable you could route uncertain cases to people and screen the rest. Ovadia and colleagues benchmarked uncertainty methods under dataset shift and found accuracy and calibration both degrade as inputs move away from training data, with "traditional post-hoc calibration does indeed fall short, as do several other previous methods". A threshold tuned on clean validation data becomes untrustworthy exactly when production drifts, which is when you are relying on it.

Automated screening belongs in the loop as a triage layer that is itself audited. It cannot be the layer that certifies the system, and an evaluation process relying solely on it has no such layer.

Why Does Adding a Human Reviewer Fail as a Control?

The evidence here is direct, and it should change how you configure a review queue.

Dratsch and colleagues showed 27 radiologists 50 mammograms alongside BI-RADS suggestions labeled as coming from an AI system. Reader accuracy "was significantly impacted by the correctness of the AI prediction of BI-RADS category". When the suggestion was wrong, accuracy fell from 79.7% to 19.8% among inexperienced readers, 81.3% to 24.8% at moderate experience, and 82.3% to 45.5% among the most experienced. Experience helped. It did not protect. The experimenters manipulated the suggestions instead of using a deployed product, so this is evidence about how people defer to a label presented as machine output. That is the reviewer-in-the-loop failure mode exactly.

Explanations do not repair it. In a randomized study of 457 clinicians, Jabbour and colleagues found correct predictions raised diagnostic accuracy 2.9 points above a 73.0% baseline while "systematically biased AI models reduced diagnostic accuracy" by 11.3 points; saliency explanations recovered only 2.3 of those points, a non-significant difference. A study of 220 physicians reading chest radiographs found saliency explanations produced the best results when the advice was correct (92.8%) and the worst when it was wrong (23.6%), while reducing the time readers spent considering the advice at all. The explanation style that helped most when the model was right hurt most when it was wrong. Specialists are not exempt: Tschandl and colleagues concluded from a skin-cancer collaboration study that "faulty AI can mislead the entire spectrum of clinicians, including experts".

Read that against Article 14 of the EU AI Act, which requires high-risk systems be designed so they can be "effectively overseen by natural persons during the period in which they are in use". Article 14(4) names the failure directly, requiring that overseers be enabled to stay aware of the tendency to over-rely on system output. What the regulation asks for is a reviewing arrangement that survives contact with a wrong prediction. Adding headcount does not meet it; workflow design does.

What Does an AI Model Evaluation Workflow Look Like Step by Step?

Six steps, each a configuration decision inside tooling most CV teams already run. Before the first, write down the success criteria: which evaluation metrics matter and at what threshold, per site or cohort. Continuous evaluation depends on metrics defined clearly up front, and those metrics should follow the business requirement the model serves.

1. Capture

Stream production frames, clips and predictions into an evaluation project continuously. Sample by risk rather than uniformly: new sites, new sensors, changed lighting or weather, cohorts with known weakness, assets real users flagged, anything a downstream system acted on automatically. The target is an evaluation dataset with test coverage of typical inputs and of the edge cases that break the model in specific scenarios.

The evidence on risk-based sampling is mixed. Munjal and colleagues, controlling for stochasticity, found that "AL methods show marginal or no advantage over the random sampling baseline" under strong regularization, with many published gains sitting inside metric variance. Against that, Elezi and colleagues report "up to a 7.7% improvement in mAP, or up to 82% reduction in labeling cost" on detection, from a method combining uncertainty, class robustness and auto-labeling. The practical reading: prioritizing by risk is worth doing, and the elaborate acquisition function usually is not.

2. Screen automatically

Deterministic checks first, since they need no model: geometry violations, impossible class co-occurrence, confidence collapse, frame-to-frame instability in video, transcripts longer than their audio. Then model-based scoring of output quality on what survives.

3. Audit the screener

Keep a human-labeled slice the router did not select and re-measure the screener against it on a cadence. This step exists because of the calibration result above. Traditional testing often relies on small samples and subjective pass/fail calls, so give the slice enough assets, and labeling rules explicit enough, that two reviewers would score it the same way. Run evaluations against it on every model, sensor and rubric change, the way continuous integration runs regression tests on every commit, and keep the slice and its evaluation results as an artifact you can show someone.

4. Route to experts

Low-confidence cases, high-stakes categories, and rule-versus-model disagreements go to domain reviewers. For any asset that will serve as a gold reference, blind the reviewer to the prediction. After Dratsch, showing it first changes the answer and invalidates the agreement statistics computed from it.

5. Correct, don't just score

A score says something is wrong. A redrawn mask, a corrected box or a fixed transcript says what right looks like. Reviewers produce the correction and tag a failure category from a controlled vocabulary. That triple — prediction, expert correction, category — is the deliverable.

6. Feed back

Corrections become evaluation data (new test cases for the next regression run), rubric revisions and training data. Track inter-reviewer agreement and failure-category mix over time: a shift in category mix tells you what changed, where an aggregate metric only tells you that something did.

Where Does the Labeling Platform Fit Each Step?

If your annotation platform already handles ontologies, reviewer roles, consensus and quality analytics, most of this is configuration rather than procurement.

Taking Kili as the worked example, strictly per documented capability: projects cover images, video, PDF, text and geospatial assets, with audio available as a beta modality on request, which is steps 1 and 5 for both modalities discussed here. Review workflows are configurable, with predefined roles at organization and project level, giving step 4 its separation of duties. Consensus measures agreement between annotators working on the same asset, and consensus labeling is blind, so annotators can't see each other's work. Honeypot scores labelers against known answers from a trusted ground-truth reference; together they supply what steps 3 and 6 depend on. Issues attach to individual annotations (bounding boxes, polygons and named entities), so a reviewer records that one box is wrong instead of flagging the whole image. Plugins allow programmatic QA, which is where step 2 lives.

Model-based pre-annotation is documented too, and it is the same mechanism the automation-bias studies indict. That argues for using it deliberately. Pre-annotate the routine bulk where throughput is the constraint; blind the gold slice where independence is the point. Kili makes that distinction at import: predictions loaded as PREDICTION labels are shown to labelers to confirm or edit, while INFERENCE labels stay hidden during labeling and are used to benchmark the model against expert ground truth.

‍

What no platform does is adjudicate. The reviewer decides, and any vendor offering fully automated evaluation is selling the failure mode described two sections above with a dashboard on top.

Does the Same Argument Hold for Speech and Audio Models?

It holds cleanly enough that speech compresses the whole case.

Benchmark-to-production gap: OpenAI's Whisper report shows models tuned on LibriSpeech reaching 2.7% word error rate on test-clean and 24.5% on the accented-speech Artie set, where Whisper scored 6.2%, noting that "supervised LibriSpeech models make roughly twice as many errors as a human". Low benchmark WER did not transfer.

Failures concentrating on populations: across five commercial systems on matched interview audio, Koenecke and colleagues found "All five ASR systems exhibited substantial racial disparities in transcription accuracy." Aggregate WER was 0.35 for Black speakers against 0.19 for white speakers. One aggregate metric hid a doubling.

Fabrication no accuracy metric catches: testing Whisper on AphasiaBank recordings in 2023, the same lead author reported that "roughly 1% of audio transcriptions contained entire hallucinated phrases or sentences" never spoken, 38% of which carried explicit harms, with rates higher for speakers with aphasia than controls.

And the reference standard has its own error rate. IBM's re-measurement of professional transcription found they "could only get down to 5.1% for the best transcriber after quality checks" on Switchboard. Human performance is a number that responds to instructions and review, which is where the annotation literature arrives from the other direction: comparing professional annotators against crowdworkers on 14,040 images, Rädsch and colleagues found "professional annotators constantly outperform Amazon Mechanical Turk crowdworkers", and that adding exemplar images to instructions cut the odds of a severe error to 0.37 times baseline while longer text instructions did nothing.

Your reference data has a quality level, and it responds to investment.

What Do You Get From Treating Corrections as Data?

Three returns, compounding at different rates.

The corpus comes first, and it is the one competitors cannot copy. A structured set of expert corrections is an eval set tracking your real failure distribution, a source of rubric language written by people who know the domain, and training data, all at once. Most teams build these separately, inconsistently, from different vintages of the same disagreements.

Second is a drift signal with content. Failure-category mix over time distinguishes a new lighting condition from a new object class from a labeling-policy change, and that is what turns guesses into informed decisions about whether the next fortnight goes to data collection, model work, or rewriting the ontology.

The third arrives as a byproduct. Article 72 of the EU AI Act requires providers of high-risk systems to run a documented post-market monitoring system that will "actively and systematically collect, document and analyse relevant data" across the system's lifetime. That lands on perception teams specifically, because Annex III classifies remote biometric identification, biometric categorisation and "AI systems intended to be used for emotion recognition" as high-risk: vision and speech systems inherit the monitoring duty and the Article 14 oversight duty directly. NIST's AI Risk Management Framework points the same way through MEASURE, asking that "The functionality and behavior of the AI system … are monitored when in production." In medical devices, FDA's guidance on predetermined change control plans says "FDA anticipates that manufacturers will monitor their device's safety … and effectiveness … over time" as authorized modifications ship. ISO 21448, the automotive SOTIF standard, treats functional insufficiency as a hazard class for systems where "situational awareness is derived from complex sensors and processing algorithms".

A running loop with reviewer identities, timestamps, agreement statistics and audit slices produces that evidence as ordinary operating output. Teams that build governance artifacts separately from the evaluation pipeline keep two accounts of one system, and the accounts diverge.

Conclusion

The build/monitor split was an artifact of how tools were sold. Annotation platforms were sold for training data, monitoring was sold as observability, and expert judgment was left with nowhere to live after launch.

Closing that gap makes expert reviewer time a permanent line item, because a perception model in production is a system under continuous measurement. New models now arrive roughly monthly, which makes model selection a recurring evaluation problem. The teams positioned for the next architecture jump are the ones already running the loop, because a corrected-output corpus specific to their sites, sensors and edge cases transfers to a new model in a way a benchmark score never does. Migration becomes an evaluation run rather than a rebuild.

Start with the twenty failures you already know about.

Resources

Production Drift and Domain Shift

• Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs – Zech et al., 2018. Internal AUC 0.931 vs 0.815 at an unseen hospital; model learned site confounders.
  • https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1002683
• External validation of deep learning algorithms for radiologic diagnosis: A systematic review – Yu, Mohajer & Eng, 2022. 81% of 86 studies reported external performance drops.
  • https://pubs.rsna.org/doi/10.1148/ryai.210064
• Benchmarking robustness in object detection: Autonomous driving when winter is coming – Michaelis et al., 2019. Detectors retain 30–60% of performance under common corruptions.
  • https://arxiv.org/abs/1907.07484

Benchmark Validity

• Do ImageNet classifiers generalize to ImageNet? – Recht et al., 2019. Faithfully reproduced test sets still cost 11–14% on ImageNet.
  • https://arxiv.org/abs/1902.10811
• Measuring robustness to natural distribution shifts in image classification – Taori et al., 2020. 204 models, 213 conditions; benchmark robustness fails to transfer.
  • https://arxiv.org/abs/2007.00644
• ObjectNet: A large-scale bias-controlled dataset – Barbu et al., 2019. Randomizing viewpoint and background cut detector performance 40–45%.
  • https://papers.nips.cc/paper_files/paper/2019/hash/97af07a14cacba681feacf3012730892-Abstract.html

Annotation Quality and Ground Truth

• Pervasive label errors in test sets destabilize machine learning benchmarks – Northcutt, Athalye & Mueller, 2021. At least 6% of ImageNet validation labels are wrong; rankings shift.
  • https://arxiv.org/abs/2103.14749
• Benchmarking object detectors with COCO: A new path forward – Singh et al., 2024. COCO-2017 mask errors penalized models predicting sharper masks.
  • https://arxiv.org/abs/2403.18819
• Inter-observer variability of manual contour delineation of structures in CT – Joskowicz et al., 2019. 5–57% volume overlap variability across observer pairs.
  • https://doi.org/10.1007/s00330-018-5695-5
• Labelling instructions matter in biomedical image analysis – Rädsch et al., 2023. Professionals beat crowdworkers; exemplar images cut severe errors to 0.37× odds.
  • https://doi.org/10.1038/s42256-023-00625-5

Limits of Automated Evaluation

• MLLM-as-a-Judge: Assessing multimodal LLM-as-a-judge with vision-language benchmark – Chen et al., 2024. Judges hold up on pairwise comparison, diverge on absolute scoring.
  • https://arxiv.org/abs/2402.04788
• Pseudo-labeling and confirmation bias in deep semi-supervised learning – Arazo et al., 2020. Training on own errors raises confidence in those errors.
  • https://arxiv.org/abs/1908.02983
• Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift – Ovadia et al., 2019. Calibration degrades under shift; post-hoc calibration falls short.
  • https://arxiv.org/abs/1906.02530

Automation Bias in Expert Review

• Automation bias in mammography: The impact of AI BI-RADS suggestions on reader performance – Dratsch et al., 2023. Accuracy fell from ~80% to ~20% on incorrect suggestions.
  • https://doi.org/10.1148/radiol.222176
• Measuring the impact of AI in the diagnosis of hospitalized patients – Jabbour et al., 2023. Biased predictions cost 11.3 points; saliency explanations recovered 2.3.
  • https://doi.org/10.1001/jama.2023.22295
• Care to explain? AI explanation types differentially impact chest radiograph diagnostic performance – Prinster et al., 2024. 92.8% with correct advice, 23.6% with incorrect.
  • https://doi.org/10.1148/radiol.233261
• Human–computer collaboration for skin cancer recognition – Tschandl et al., 2020. Faulty AI degraded performance at every expertise level.
  • https://doi.org/10.1038/s41591-020-0942-0

Risk-Based Sampling

• Towards robust and reproducible active learning using neural networks – Munjal et al., 2022. Gains over random sampling are inconsistent and often inside metric variance.
  • https://arxiv.org/abs/2002.09564
• Not all labels are equal: Rationalizing the labeling costs for training object detection – Elezi et al., 2022. Up to 7.7% mAP gain or 82% labeling cost reduction.
  • https://arxiv.org/abs/2106.11921

Speech and Audio

• Robust speech recognition via large-scale weak supervision – Radford et al., 2022. LibriSpeech 2.7% vs 24.5% on accented speech for the same class of model.
  • https://cdn.openai.com/papers/whisper.pdf
• Racial disparities in automated speech recognition – Koenecke et al., 2020. Aggregate WER 0.35 for Black speakers vs 0.19 for white speakers.
  • https://www.pnas.org/doi/10.1073/pnas.1915768117
• Careless Whisper: Speech-to-text hallucination harms – Koenecke et al., 2024. ~1% of transcriptions contained fabricated phrases; 38% carried explicit harms.
  • https://dl.acm.org/doi/10.1145/3630106.3658996
• English conversational telephone speech recognition by humans and machines – Saon et al., 2017. Professional human WER of 5.1% on Switchboard after quality checks.
  • https://www.isca-archive.org/interspeech_2017/saon17_interspeech.html

Governance and Standards

• Regulation (EU) 2024/1689, Article 14: Human oversight – European Union, 2024. Design obligation for effective oversight throughout deployment.
  • https://artificialintelligenceact.eu/article/14/
• Regulation (EU) 2024/1689, Article 72: Post-market monitoring – European Union, 2024. Documented monitoring system collecting performance data across the lifetime.
  • https://artificialintelligenceact.eu/article/72/
• Regulation (EU) 2024/1689, Annex III – European Union, 2024. Biometric identification, categorisation and emotion recognition as high-risk.
  • https://artificialintelligenceact.eu/annex/3/
• AI Risk Management Framework (AI RMF 1.0) – NIST, 2023. GOVERN, MAP, MEASURE, MANAGE; production monitoring under MEASURE.
  • https://nvlpubs.nist.gov/nistpubs/ai/nist.ai.100-1.pdf
• Predetermined change control plans for AI-enabled device software functions – FDA, 2025. Expectation of ongoing safety and effectiveness monitoring.
  • https://www.fda.gov/media/166704/download
• ISO 21448:2022, Road vehicles — Safety of the intended functionality – ISO, 2022. Functional insufficiency as a hazard class for sensor-driven perception.
  • https://www.iso.org/standard/77490.html

Kili Technology

• Consensus overview – Kili Technology documentation, on measuring agreement between annotators labeling the same asset.
  • https://docs.kili-technology.com/docs/consensus-overview
• Kili plugins – Kili Technology documentation, on Python business rules that run automatically on each submitted or reviewed label.
  • https://docs.kili-technology.com/docs/kili-plugins
• Importing labels and pre-annotations – Kili Python SDK documentation, on PREDICTION labels shown to labelers versus INFERENCE labels hidden for benchmarking.
  • https://python-sdk-docs.kili-technology.com/26.1/sdk/tutorials/importing_labels/
• Kili Technology platform – Annotation and multi-step review workflows for image, video, document and geospatial data.
  • https://kili-technology.com/