Data Story: Decision Models, Calibrated Judges and the Labels They're Calibrated Against

Introduction

Between mid-September and late October 2026, a new kind of model went from one startup's launch to a product category. TypeSafe AI released Jev on 15 September. By 9 October it had raised about $870 million at a $7.5 billion valuation, according to Bloomberg. Liquid AI, OpenAI, Cloudflare and AWS followed with their own versions. Microsoft announced Decision-1 on 9 October. Three days later, a team from StartLux, the University of Hong Kong and SUSTech published the first detailed technical paper on how to train one.

These are decision models. Instead of generated text, they return structured outputs. You give them an input and a set of predefined options, and they return a probability for each option in a single pass. The pitch is that the probability is calibrated. That is what makes them useful as LLM judges (large language models that grade other models' outputs), verifiers and routers. Microsoft states the target plainly: on representative cases, "a 90% prediction should be right about nine times out of 10."

Model calibration is a relationship between a model's probabilities and a set of labels. "Right nine times out of ten" means right according to some reference answers, and someone produced those answers. So the useful question about a calibrated judge is what it was calibrated against. For the models launched this autumn, there are three answers. One is a benchmark whose answers a program can check. Another is the averaged opinion of two frontier LLMs. The rest publish nothing at all.

The sections below check what each launch says about its labels, then what the research shows happens to calibration when labels are noisy or contested. The last section is a protocol for calibrating a decision model against your own experts.

What Is a Decision Model, and Why Did So Many Launch at Once?

A decision model answers a typed question about an input and returns a probability distribution over the candidate answers. The question can be yes/no, a choice among options, a score on an ordered scale, or a rubric item. A decision model evaluates LLM outputs, agent steps or incoming requests against defined criteria, and gives structured answers rather than prose. There is no free-form text. In Decision-1's case there is no natural language explanation either: InfoWorld notes that it "returns scores but does not provide a rationale for each decision." Downstream software reads the probability and acts on it: accept the output, escalate to a more capable model, or send the case to a person.

In that sense these models work like classification models with a language model's reading ability. The Register put it more bluntly: "Basically, Jev is a classifier with brains."

The economics explain the timing. Microsoft prices Decision-1 at $0.042 per million input tokens, with output tokens free. The Next Web reported the same price and noted that Decision-1 is built on Alibaba's Qwen3.5-9B. TypeSafe charges the same rate for Jev, with output also free. InfoWorld worked through Microsoft's own example. An incident-prioritisation request with 86 input tokens cost about $0.0000036, so "$1 would cover more than 275,000 similar decisions" (Microsoft's figures). At that cost, and with one forward pass per decision instead of several LLM calls, a judge call stops being something you sample. For teams evaluating LLM applications, it becomes something you can run on every agent step and every production output.

InfoWorld's survey of the category describes vendors carving decision-making out as a separate model layer for agents. Some are open weights or open-source models. AWS released Strands Decider 2B as open source, StartLux published its weights, and Baseten put mammoth-1 and elephant-1 on Hugging Face. Others, including Decision-1 and Liquid AI's d1, are hosted APIs. Community projects have started ranking open-weight decision models against each other. Fortune covered how fast Jev was adopted and how quickly OpenAI followed.

The use cases the vendors list are evaluation use cases. TypeSafe's launch post pitches Jev with the line "Score, judge, verify, guardrail, and detect jailbreaks." Microsoft lists rubric-based assessment of AI responses and agent actions. The decision model is being sold as a faster, cheaper judge model whose numbers you can threshold. Teams write custom judges as a scoring rubric or a list of options, and the model returns a probability for each one.

How StartLux trains one

The StartLux-Decision paper (arXiv preprint, vendor-authored) is the most detailed public account. It trains a model family from 0.8B to 27B parameters in three stages:

1. Supervised fine-tuning on about 2.7 million examples (0.8 million for the 27B model). The targets are soft distributions, meaning probabilities spread across the answers, rather than single answers.
2. Adapter-based decision training on about 1.7 million more examples.
3. Reinforcement learning (RL) with a method the authors call RLCD². Its reward is a proper scoring rule, a score that is best when the model reports its honest probability (explained below).

The paper also introduces "AutoThink" variants, which answer directly when confident and reason at inference time only when their first probability falls below a 0.9 gate. On the authors' own DecisionPivot benchmark, AutoThink beats the single-pass models by 11.5 to 18.4 points on pair accuracy. For the 9B model, RL cut average reasoning length from 2,494 tokens to 868.

How Did LLM-Based Evaluation Work Before Decision Models?

Traditional metrics such as BLEU and ROUGE compare generated text with a reference text by counting overlapping words. The authors of G-Eval (EMNLP 2023) found they have "relatively low correlation with human judgments, especially for open-ended generation tasks." LLM-based evaluation replaced word matching with a capable model reading the LLM generations themselves. Evaluation frameworks package each test case as an input, sometimes with an expected output. For retrieval systems, the test case also holds the retrieved context.

Zheng et al. (NeurIPS 2023) describe the three set-ups still in use:

• Pairwise comparison: the judge sees two answers to the same input and picks the better one, or declares a tie.
• Single answer grading: the judge scores one output at a time on a scoring scale.
• Reference-guided grading: the judge compares the output with a reference solution.

The judge is asked about quality dimensions such as helpfulness. The authors note that "within helpfulness, there are multiple dimensions like accuracy, relevance, and creativity." They reported that GPT-4 as a judge reached "over 80% agreement" with human evaluation, the same level as agreement between humans. Most production judging is reference-free judging, though: there is no expected output, only an evaluation prompt and evaluation criteria.

Those judges have known biases. Zheng et al. define position bias as a judge favouring "certain positions over others." Their fix is to call the judge twice with the answers swapped and count a win only when both orders agree. A comprehensive survey of LLM-as-a-Judge catalogues the wider set of strategies, from improving consistency to mitigating biases.

G-Eval went one step further. Besides chain-of-thought evaluation steps (the model writes out its reasoning before scoring), it used "the probabilities of output tokens" to weight its scores. It reached a Spearman rank correlation of 0.514 with human ratings on summarisation. A written answer such as "8 out of 10" says nothing about how sure the judge is. Decision models turn one idea into a product: read the model's confidence from its probabilities, not from the LLM scores or judge scores it writes out.

What Does Model Calibration Mean for an LLM Judge?

A model is calibrated when its confidence matches its accuracy. The scikit-learn documentation gives the plain version: of the samples a well-calibrated classifier scores around 0.8, about 80% should belong to the positive class. Guo et al. (ICML 2017) showed that modern deep networks, unlike older shallow ones, are usually overconfident. A one-parameter fix called temperature scaling, which softens all the probabilities by the same factor, corrects much of it.

Three measurements come up whenever a decision model claims calibration:

• Expected calibration error (ECE) sorts predictions into confidence bins and averages the gap between each bin's confidence and its accuracy. StartLux reports this kind of gap, in percentage points, over 15 bins.
• The Brier score is the mean squared difference between the predicted probability and the outcome (1 if the label says yes, 0 if not). Lower is better.
• Log loss penalises confident wrong answers heavily.

The Brier score and log loss are proper scoring rules. Gneiting and Raftery (JASA, 2007) define the property: a rule is strictly proper if a forecaster gets the best expected score only by reporting their true belief. That is why StartLux uses one as an RL reward. It removes any incentive for the model to shade its probabilities.

Every one of these formulas needs a reference. Accuracy per bin needs a correct answer. The Brier score needs an outcome. Log loss needs a label. Model calibration is never a property of the model alone; it is a property of the model and a reference label set together. When a vendor says its judge is calibrated, the claim is incomplete until it names that set. Calibration is a measurement problem about the labels as much as about the model.

This matters more for judges than for chat models. The GPT-4 technical report found that post-training "hurts calibration significantly." On a subset of MMLU, a multiple-choice knowledge benchmark, expected calibration error rose from 0.007 for the pre-trained model to 0.074 after RLHF (reinforcement learning from human feedback). This is an OpenAI self-report. Decision models exist partly to win that calibration back. What they are calibrated back towards is the open question.

What Are This Autumn's Decision Models Actually Calibrated Against?

The launch materials answer this unevenly. Here is what each source states about its reference labels.

StartLux: calibrated on answers a program can check

StartLux is the only one to publish calibration numbers. Its AutoThink models show gaps of 2.4 points (4B), 3.3 (9B) and 5.2 (27B). The hosted version of Jev, run as a baseline, shows 13.8, and Cloudflare's Clef 19.3. Those numbers come from DecisionPivot, which the authors generated from 13 task families of 600 items each and describe as "a programmatically verifiable benchmark of computational decisions." Every answer can be checked by a program.

That is a sensible choice for a controlled test. It also leaves out most real judge work, where no program can check the answer and a person has to decide.

The training data is more mixed. The paper's appendix codes each source as "human annotation or source gold label," model-generated labels, or programmatic labels. The model-generated portion includes teacher-labelled examples and API-generated examples kept "only when two independent verifier models agree with their labels." The teacher, the API and the verifiers aren't named. The paper also doesn't say how its soft target distributions were built. It doesn't discuss annotator disagreement at all.

The headline 73.4 score comes from the public portion of a community-run leaderboard. The authors flag the limit themselves. They used that public subset to choose their recipes, so "these scores reflect model selection and are not independent test results."

Jev: calibrated against two frontier models

TypeSafe's launch post says it uses "the average of GPT-6 Astra and Fable 5.1 as the reference answer" for evaluation, and adds that this "biases answers towards OpenAI and Anthropic's models." Elsewhere on the page, TypeSafe says it doesn't optimise for a ground-truth classification. A Jev probability of 0.9, then, estimates how strongly two other models would agree with that answer.

Decision-1: the claim without the measurement

Microsoft describes the property and stops there. The announcement has no reliability diagram, no ECE and no Brier score. There are no independent evaluations either: InfoWorld notes the results "have not been independently verified." The Foundry model card describes the training data only as public datasets plus synthetic data. Microsoft's own deployment documentation is more cautious than the launch post. It says "Calibration is strongest on familiar task types." It warns that scores change with the phrasing and order of the options. And it advises users to "Prefer scores for relative ordering and thresholds rather than as absolute, calibrated ratings." It also tells them to validate any threshold "against labeled examples before you automate a gate." InfoWorld's report on Decision-1 quotes Kanerika's Amit Kumar Jena on the risk: "a precise-looking probability can give an uncertain decision more authority than it deserves."

OpenAI's Decisions API runs on GPT-6 Luna, listed as "GPT-6 Luna Decisions" in Microsoft's comparison. OpenAI's Decisions documentation goes furthest in handing the problem back: "Use labeled examples from your application to set thresholds." Neither the guide nor its API reference makes a calibration claim. Liquid AI's documentation calls d1's probabilities "calibrated" and publishes no metric behind the word. Baseten's launch of mammoth-1 and elephant-1 lists 6.21 million training examples, including "judging and expert domains," and no calibration metric.

The vendors' own documentation, then, leaves the reference labels to the customer.

Can an LLM Judge Be Perfectly Calibrated and Still Wrong?

Yes, and the research shows it in both directions.

Baan et al. (EMNLP 2022) built an oracle: a classifier that predicts exactly the distribution of human labels for each item in ChaosNLI, a dataset with 100 annotations per example. Scored with standard ECE against majority-vote labels, the oracle came out at 0.25. An ordinary RoBERTa text classifier scored better, at 0.14. The authors' summary: "the oracle classifier is severely miscalibrated according to ECE." A model that perfectly represents human opinion looks badly calibrated, because the reference collapses that opinion into one label. Temperature scaling then cut RoBERTa's ECE from 0.14 to 0.03. Its fit to the actual human distribution barely moved (0.26 to 0.22 on the authors' distribution-based measure). Calibration against the majority label improved; the model didn't get closer to what people think.

Noisy labels cause the opposite problem. A preprint by Hanada and Futami (October 2026) shows that annotation noise biases ECE estimates with respect to the true label. More samples or majority voting don't in general remove the bias. In their experiments, recalibrating a model towards annotator agreement can make its calibration against the true label worse. On a census-income benchmark, ECE against the original labels (×100) was 6.58 for the base model. Naive binned recalibration on the noisy annotations raised it to 9.17. The authors' noise-corrected method brought it down to 0.80. Naive temperature scaling did worse still, at 12.22.

The same logic applies to RLCD²-style training. A proper scoring rule rewards a model for reporting its true belief about the reference label. If the reference is wrong in a consistent way, the reward trains the model to reproduce that error, with confidence.

StartLux's own data shows a third limit. A 9B verifier's answers at 80–90% confidence were 92% correct when scored one at a time. After best-of-32 selection (picking the top-scored of 32 candidate answers), only 78% were correct, because selection favours the answers the verifier overrates. Calibration holds for a specific use of the model, as well as for a specific label set.

How Often Do the Reference Labels Themselves Change?

More often than the word "gold" suggests.

ChaosNLI collected 100 new annotations per item for three natural language inference benchmarks, where annotators decide whether one sentence follows from another. For SNLI and MNLI, the authors re-annotated the borderline items, where the original majority label had three of five votes behind it. With 100 votes, the majority label changed on 24.97% of those SNLI items and 31.77% of those MNLI items. αNLI's gold labels came from a single annotator, so they re-annotated its whole development set. The majority label changed on 10.64% of it. The gold label is most fragile on the borderline items, and those are the items where a judge's confidence decides what happens next.

Preference data, the kind most LLM judges and reward models learn from, shows the same pattern. Zhang et al. (2024), a NeurIPS workshop paper with authors from NVIDIA and AI2, found that human annotators diverged on 39% of MultiPref pairs and 24% of HelpSteer2 pairs. Bradley-Terry reward models, the standard way to turn pairwise preferences into scores, were just as confident on those contested pairs as on agreed ones: an average P(chosen > rejected) of 0.798 versus 0.786. Their ability to flag the contested pairs was below chance (AUROC 0.458 to 0.482, where 0.5 is a coin flip). LLM judges still picked a winner on 73.8% of them.

Who labels matters as much as how many. Nahum et al. (EMNLP 2025), from the Technion and Google Research, re-labelled factual-consistency benchmarks and found 6% to 21% label errors in the originals. On the same items, crowd workers on MTurk reached a Fleiss κ of 0.074, barely above chance. (Fleiss κ is a chance-corrected agreement score: 0 means chance level and 1 means perfect agreement.) Experts reached 0.486 before discussing their disagreements and 0.851 after. Correcting the labels raised GPT-4's measured ROC AUC from 0.81 to 0.93 and dropped one fine-tuned model from second to eighth in the ranking.

Our earlier data story, Who Judges the Judge?, covered the same split in LLM-judge evaluation. Expert and crowd pools that were each internally unanimous still disagreed on the majority label for 23.6% of items. A decision model calibrated against crowd labels and one calibrated against expert labels are calibrated against different targets. Both can report a tidy 3-point gap.

Do Expert Labels Solve the Model Calibration Problem?

They make the target meaningful, but experts disagree too, and that disagreement is part of the signal.

In OpenAI's HealthBench, 262 physicians wrote rubric criteria for 5,000 health conversations, each a single-turn or multi-turn conversation with a model. On the consensus criteria, "both physician-physician and model-physician agreements range from 55% to 75%" (an OpenAI self-report). A medRxiv preprint by Williams et al. had six Rwandan doctors rate model answers to health queries. Their agreement with each other was a quadratic-weighted κ of 0.32. The AI judges agreed with each other more than the doctors did (0.53), and with the doctors less (0.22).

Say your experts agree on 70% of items. A judge then cannot meaningfully claim to be right 95% of the time against "the expert answer," because on the remaining items there isn't one. Expert agreement is the ceiling on any calibration claim. It has to be measured before the judge is.

Two lines of work show what to do with the disagreement instead of discarding it:

• Keep the distribution. Peterson et al. (ICCV 2019) collected about 50 human labels per CIFAR-10 test image and trained on the full distribution of answers. The resulting models were "significantly less confident when incorrect" than ones trained on single labels. If several experts label each calibration item, the spread of their answers is a better target for a probability than a majority vote. Our guide to model distillation covers the same idea from the teacher side, as soft targets.
• Model each expert. Microsoft researchers' LLM-Rubric (ACL 2024) calibrated a judge to each of 24 trained annotators separately. The reason: "the humans do not fully agree with one another," and each used the rating scale differently.

How Should You Calibrate a Decision Model Against Your Own Experts?

The vendors have made the judge cheap. What they leave to you is the reference set. The following steps extend the validation protocol from Who Judges the Judge? from agreement to calibration.

1. Name the reference set

Before reading any calibration number, write down who produced the labels it was measured against, under which rubric version, and how disagreements were resolved. Relying solely on a vendor's calibration claim means relying on labels you haven't seen. For a vendor model, that is the question to send to the vendor. For your own calibration set, it is the first line of the methods note.

2. Measure your experts before you measure the judge

Run the calibration items past several experts independently, then compute their agreement, including a chance-corrected figure. Our guide to building a custom AI benchmark recommends a 50 to 100 item gold set and at least 75% agreement between judge and expert consensus. Who Judges the Judge? adds that the figure should be read alongside a chance-corrected statistic. That floor still applies. Score each expert against a set of items with a trusted answer, too, so that one outlier doesn't shift the target.

3. Keep every expert's label

Collapse to a majority vote only when you have to. The share of experts choosing each option is the closest thing you have to a "true" probability for that item. Baan et al. show that a standard ECE hides it.

4. Size the set for calibration, not just agreement

An agreement check asks one question of the whole set. A calibration check asks it again inside every confidence bin, and each bin needs enough items to estimate its own accuracy. A set large enough to validate agreement is usually too small for a reliable reliability diagram (the chart of confidence against accuracy, bin by bin). Plan a larger calibration set, and grow it from the cases where the judge and the experts disagree.

5. Set thresholds for the way the judge will be used

StartLux's best-of-32 result shows that calibration measured on single answers doesn't hold once you select among many. Ashish Chaturvedi of HFS Research, quoted by InfoWorld, adds that "thresholds tuned for one model will not transfer to another." Set confidence thresholds on data that reflects the deployment. A method such as Trust or Escalate (ICLR 2025) sets a judge's confidence cut-off so that agreement with human labels meets a level the user specifies. Our piece on human-in-the-loop and human-on-the-loop validation covers how to route escalated cases to manual review.

6. Recalibrate when anything changes

A new model version, a reworded rubric, a reordered option list or a new expert pool all change the relationship between the probability and the labels. Microsoft's documentation says as much about phrasing and order.

Where Kili fits

Kili covers this part of the workflow. Teams import the model's outputs as text assets and upload the decision model's verdicts as predictions through the API or SDK. Human reviewers then confirm or correct them. Consensus, enabled on the labeling step, has several experts label the same asset blind and measures their agreement. Honeypot, enabled on one workflow step, scores each expert's work against a trusted ground-truth reference. Enforce Step Separation, on by default, means no one reviews their own work. The export can include the latest label from each author on each asset, which keeps the per-expert spread that step 3 calls for. Our guide to golden datasets covers how those reference sets are built.

Conclusion: The Label Set Is the Product

Decision models change the cost structure of evaluation. A judge call priced at a few cents per million tokens can sit on every agent step and score every RL rollout. A calibrated probability can automatically score each case and decide which ones reach human judgment. That makes the probability more consequential, and its reference labels with it.

When a decision model is used as a reward, the consequences compound. Our RLVR explainer covers how learned graders get gamed, and the MiMo-V2.6 data story showed how much RL compute now goes to the grader. Train a decision model against crowd labels, or against two other LLMs, then use it to train a policy. It passes that reference on to every model it grades.

The vendors are converging on the same architecture, the same price and the same proper scoring rules. What will separate a trustworthy judge from a confident one is the set of labels it was calibrated against, and whether anyone measured the people who made them.

Resources

Decision models: papers, launches and documentation

• StartLux-Decision: AutoThink Decision Models with Reinforcement Learning for Calibrated Deliberative Decisions – Zhuang et al., 2026, arXiv preprint (vendor-authored: StartLux)
  • https://arxiv.org/abs/2610.17309
• Introducing Microsoft-Decision-1, our model for fast decision-making – Microsoft Command Line, 9 Oct 2026 (self-reported results)
  • https://commandline.microsoft.com/microsoft-decision-1-model-foundry/
• Deploy and use Microsoft-Decision-1 in Microsoft Foundry – Microsoft Learn
  • https://learn.microsoft.com/en-us/azure/foundry/foundry-models/how-to/use-foundry-models-microsoft-decision
• Microsoft-Decision-1 model card – Microsoft Foundry
  • https://ai.azure.com/catalog/models/Microsoft-Decision-1
• Introducing System One models & Jev – TypeSafe AI, 15 Sep 2026
  • https://typesafe.ai/blog/introducing-system-one-models-and-jev
• Decisions – OpenAI API documentation
  • https://developers.openai.com/api/docs/guides/decisions
• Decision models – Liquid AI documentation (d1)
  • https://docs.liquid.ai/lfm/models/decision-models
• The top ranked decision models in the world: mammoth-1 and elephant-1 – Baseten, 21 Oct 2026 (self-reported results)
  • https://www.baseten.co/blog/the-best-decision-models-in-the-world-mammoth-1-and-elephant-1/

Coverage

• Andreessen Horowitz backs Jev maker at $7.5 billion value – Bloomberg, 9 Oct 2026
  • https://news.bgov.com/private-equity/andreessen-horowitz-backs-jev-maker-at-7-5-billion-value
• Jev, an AI for making quick decisions, has been a viral hit in Silicon Valley. But OpenAI is hot on its heels – Fortune, 8 Oct 2026
  • https://fortune.com/2026/10/08/jev-an-ai-for-making-quick-decisions-has-been-a-viral-hit-in-silicon-valley-but-openai-is-hot-on-its-heels/
• As AI agents grow, vendors carve out decision-making as a separate model layer – InfoWorld, 9 Oct 2026
  • https://www.infoworld.com/article/4233075/as-ai-agents-grow-vendors-carve-out-decision-making-as-a-separate-model-layer.html
• Microsoft separates decision-making from generative AI with a new model – InfoWorld, 13 Oct 2026
  • https://www.infoworld.com/article/4234290/microsoft-separates-decision-making-from-generative-ai-with-a-new-model.html
• Microsoft launches a decision model built on Alibaba's Qwen – The Next Web, 12 Oct 2026
  • https://thenextweb.com/news/microsoft-decision-1-qwen-decision-model
• Shut up and calculate: Jev's new AI primitives for coders – The Register, 23 Sep 2026
  • https://www.theregister.com/a/5298431

LLM-as-a-judge methods

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena – Zheng et al., NeurIPS 2023 Datasets and Benchmarks
  • https://arxiv.org/abs/2306.05685
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment – Liu et al. (Microsoft), EMNLP 2023
  • https://arxiv.org/abs/2303.16634
• A Survey on LLM-as-a-Judge – Gu et al., 2024, arXiv preprint
  • https://arxiv.org/abs/2411.15594

Calibration and scoring rules

• On Calibration of Modern Neural Networks – Guo et al., ICML 2017
  • https://proceedings.mlr.press/v70/guo17a.html
• Strictly Proper Scoring Rules, Prediction, and Estimation – Gneiting & Raftery, JASA 2007
  • https://doi.org/10.1198/016214506000001437
• Comparison of calibration of classifiers – scikit-learn documentation
  • https://scikit-learn.org/dev/auto_examples/calibration/plot_compare_calibration.html
• GPT-4 Technical Report – OpenAI, 2023, arXiv (self-reported)
  • https://arxiv.org/abs/2303.08774
• Stop Measuring Calibration When Humans Disagree – Baan et al., EMNLP 2022
  • https://aclanthology.org/2022.emnlp-main.124
• Noisy-Label Calibration with Multiple Unreliable Labels – Hanada & Futami, 2026, arXiv preprint
  • https://arxiv.org/abs/2610.19543
• Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement – Jung, Brahman & Choi, ICLR 2025
  • https://arxiv.org/abs/2407.18370

Label disagreement: crowd, experts and preferences

• What Can We Learn from Collective Human Opinions on Natural Language Inference Data? (ChaosNLI) – Nie, Zhou & Bansal, EMNLP 2020
  • https://aclanthology.org/2020.emnlp-main.734
• Diverging Preferences: When do Annotators Disagree and do Models Know? – Zhang et al., 2024, NeurIPS 2024 Pluralistic Alignment Workshop
  • https://arxiv.org/abs/2410.14632
• Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance – Nahum et al., EMNLP 2025
  • https://aclanthology.org/2025.emnlp-main.1360.pdf
• HealthBench: Evaluating Large Language Models Towards Improved Human Health – Arora et al., OpenAI, 2025, arXiv (self-reported)
  • https://arxiv.org/abs/2505.08775
• Human Evaluators vs. LLM-as-a-Judge: Toward Scalable, Real-Time Evaluation of GenAI in Global Health – Williams et al., 2025, medRxiv preprint
  • https://doi.org/10.1101/2025.10.27.25338910
• Human Uncertainty Makes Classification More Robust – Peterson et al., ICCV 2019
  • https://arxiv.org/abs/1908.07086
• LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts – Hashemi et al. (Microsoft), ACL 2024
  • https://arxiv.org/abs/2501.00274

From Kili

• Who Judges the Judge? – data story on LLM-judge validity and annotator pools
  • https://kili-technology.com/blog/data-story-who-judges-the-judge-llm-as-a-judge
• How to Build a Custom AI Benchmark – gold sets, agreement thresholds and rubric design
  • https://kili-technology.com/blog/how-to-build-a-custom-ai-benchmark
• How to Build Golden Datasets for Testing, Fine-Tuning and Evaluating AI Models
  • https://kili-technology.com/blog/how-to-build-golden-datasets-for-testing-fine-tuning-and-evaluating-ai-models
• Human-in-the-Loop, Human-on-the-Loop and LLM-as-a-Judge for Validating AI Outputs
  • https://kili-technology.com/blog/human-in-the-loop-human-on-the-loop-and-llm-as-a-judge-for-validating-ai-outputs
• Model Distillation Explained
  • https://kili-technology.com/blog/model-distillation
• RLVR Explained
  • https://kili-technology.com/blog/rlvr-reinforcement-learning-with-verifiable-rewards
• MiMo-V2.6 Data Story
  • https://kili-technology.com/blog/data-story-mimo-v2-6-llm-as-a-judge
• Talk to the Kili team
  • https://kili-technology.com/contact