LLMs
AI Evaluation

Reward Models Explained: Preference Data, Rubric Rewards and How They Fail

A reward model turns human judgment into a training score. Learn how preference and rubric reward models work in law, medicine and insurance, and how they fail.

Kili Technology
·
Nov 9, 2026

Table of contents

Heading2
Heading3

AI Summary

  • Benchmark accuracy only loosely predicts the policy a reward model trains.
  • Reward models turn annotator disagreement into confident score gaps.
  • Human rubrics beat model-written ones by 22.1 to 28.3 points (RubricBench).
  • Physician-to-physician agreement on HealthBench ranged from 55% to 75%.
  • We found no published insurance reward model study.

Introduction

On 13 October 2026, two preprints about reward models went up on arXiv. The first, ReVeR, builds a rubric-based reward model and tests it against two earlier rubric reward models. One of those baselines was the more accurate judge of human preferences, at 66.3% against 63.9%. Each then supplied the reward for GRPO, a reinforcement learning method, to train the same policy model (the model being optimised). The more accurate one left the policy worse than where it started (51.8 against a 53.9 baseline). The less accurate one improved it (55.1).

The second, LegalRewardBench, asked how well reward models judge answers to legal questions, mostly under Chinese law. The best general-purpose model it tested agreed with the human-verified preferences 86.85% of the time. The best reward models fine-tuned for the job reached 78.25% and 78.14%.

A reward model is a measurement instrument. It stands in for a person's judgment every time a model is trained, filtered or ranked. It is only as good as the judgment it was built from and the domain it was tested in. This guide explains what a reward model is and how preference data and rubric rewards produce one. It then covers what changes in law, medicine and insurance, and the ways reward models fail before any policy learns to game them. Gaming itself is called reward hacking. Our RLVR guide, on reinforcement learning with verifiable rewards, covers it, so this article links there rather than repeating it.

What Is a Reward Model?

A reward model is a model trained to score a response the way a human judge would, so that the score can stand in for human feedback during training. In reinforcement learning from human feedback (RLHF), the policy model generates answers and the reward model scores them. Training then updates the policy to raise the score. It is one of the main ways labs align models with human preferences. Our 2026 RLHF guide walks through that pipeline stage by stage.

Classic reinforcement learning takes its reward function from the environment: points in a game, a robot reaching its target. Language generation has no such score for a good answer, so the reward has to be learned from people. Nathan Lambert's chapter on reward modeling notes that reward models have long served "as a proxy for environment rewards". He calls the practice "closely related to inverse reinforcement learning", which infers a reward function from observed behaviour.

The idea predates large language models. Deep Reinforcement Learning from Human Preferences, presented at Neural Information Processing Systems (NeurIPS) 2017, learned a reward from people's choices between pairs of short trajectory segments. With it, agents learned Atari games and simulated robot locomotion "without access to the reward function".

Three variants matter for this guide:

  • Outcome reward models and process reward models. An outcome reward model scores the final answer. Training Verifiers to Solve Math Word Problems trained verifiers to judge whether a solution reached the correct answer. At test time, the system picked the highest-ranked of many candidate solutions. In that setup, the authors report a gain about the size of a 30x increase in model size. A process reward model scores each step of the reasoning. OpenAI's Let's Verify Step by Step found that step-level feedback beat outcome-level feedback on competition maths problems, and released a dataset of 800,000 step-level human labels.
  • Generative reward models. Instead of a single number, the model writes its reasoning and then a verdict. This is the same pattern as LLM-as-a-judge, where a large language model (LLM) grades another model's output. Google DeepMind's generative verifiers are trained with ordinary next-token prediction, which lets the verifier use chain-of-thought reasoning.
  • Rubric reward models. The judge scores a response against a list of written criteria. This is the family ReVeR belongs to, and the one that matters most for expert domains.

How Does a Reward Model Work?

The key idea is to turn a comparison into a number. Lambert describes the canonical reward model as "derived from the Bradley-Terry model of preference", a statistical model for ranking items from pairwise comparisons. Under the Bradley-Terry model, the probability that one response beats another depends on the difference between their scores.

The most common reward model is a transformer language model with a linear head on top. Lambert calls appending "a small linear head to the language model" the "most common way reward models are implemented". The prompt and a candidate response go in as one sequence. A forward pass runs through it, and the reward head reads the final hidden state and outputs a scalar reward score.

The starting point is usually a chat model, not a raw base model. OpenAI's InstructGPT paper, on training language models to follow instructions, built its reward model from the initial supervised model. That is the supervised fine-tuning (SFT) checkpoint "with the final unembedding layer removed": the layer that turns the model's internal state into next-word probabilities is gone. Meta's Llama 2 initialised its reward models from chat model checkpoints and swapped the head used for next-token prediction for one that outputs a scalar reward.

Reward model training is ordinary supervised machine learning. The training objective is a negative log likelihood. For each pair, the loss function subtracts the rejected response's score from the chosen response's score. A sigmoid turns that difference into a probability, and the loss is its negative log. Lambert puts it as "minimizing the expected negative log-likelihood". The objective function sees only differences between scores, never absolute values; InstructGPT notes that "the RM loss is invariant to shifts in reward". What the reward model learns, then, is a ranking.

Training is also short. Lambert writes that "the most common practice is to train for only 1 epoch to avoid overfitting", that is, one pass over the training data. Llama 2 found that "training longer can lead to over-fitting". InstructGPT explains why. Comparisons from the same prompt are highly correlated. When the team shuffled them into one dataset, "a single pass over the dataset caused the reward model to overfit". They fixed it by training all comparisons from one prompt as a single batch element.

How Is a Reward Model Trained From Preference Data?

A preference reward model learns from comparisons. People see two responses to the same prompt and say which is better (pairwise comparisons), or rank several at once (k-wise comparisons). In InstructGPT, labellers ranked between 4 and 9 responses for a given prompt, and each ranking was split into every possible pair. Those rankings trained a 6-billion-parameter reward model. The team chose 6B over 175B because it "saves a lot of compute" and because 175B training "could be unstable". The labellers agreed with each other 72.6% of the time.

That number is the ceiling on what the reward model can learn. When trained labellers agree only 72.6% of the time, the reward model is fitting a mix of shared judgment and individual taste. Nothing in the loss function tells the two apart.

Anthropic's Training a Helpful and Harmless Assistant collected human preference data the same way: crowdworkers were "shown two responses" and picked the more helpful one. The paper calls its reward model a preference model and argues that its scores "should predict the probability that humans will prefer" one response over the other.

Llama 2 shows how this plays out in production. Training reward models on Meta's own data plus open sets including Anthropic Helpful and Harmless, Meta built two separate ones, one for helpfulness and one for safety. Both are human values, and Meta found the two "sometimes trade off". The safety reward model scored the chat model's responses for safety during RLHF. Meta also added a margin term so that strong preferences count for more than weak ones. Even so, the helpfulness reward model picked the preferred response 80.7% of the time when annotators said one answer was significantly better. When they said it was negligibly better or were unsure, that fell to 54.7%. On close calls, it was barely above a coin flip.

Multi-objective reward modeling takes the split further. ArmoRM was published in Findings of Empirical Methods in Natural Language Processing (EMNLP) 2024. It trains on absolute ratings along several human-interpretable dimensions "(e.g., honesty, verbosity, safety)". A gating network then weighs those dimensions for each prompt.

Our DPO vs RLHF guide covers what makes a preference pair worth training on. That means a clear margin, a chosen answer that is good rather than merely better, and length that isn't doing the deciding. It also covers the length and sycophancy biases that preference data passes on to the model.

What Does a Reward Model Do in the RL Training Loop?

In proximal policy optimization (PPO), the RL method InstructGPT used, each step of the training loop runs the same way. The policy model writes a response, the reward model scores it, and the policy is updated toward higher scores. Two other parts keep that in check. A value function estimates how much reward to expect, and InstructGPT initialised it from the reward model. A per-token KL penalty keeps the policy close to a frozen reference model, the SFT model it started from. The penalty grows as the policy's word probabilities drift from the reference model's. InstructGPT also mixed pretraining gradients, the original language modeling objective, into the update, a variant it called PPO-ptx.

Across the training process, the policy optimises the reward model, not the human judgment behind it. Lambert's chapter says reward models "act as proxy objectives for the core optimization". When the policy finds model outputs the proxy rewards but people would reject, that is reward hacking.

Direct alignment algorithms such as direct preference optimization (DPO) drop the separate reward model and train the policy on the preference pairs directly. The DPO vs RLHF guide above covers what that changes. The preference data, and the people who produced it, stay the same.

How Do You Know if a Reward Model Is Any Good?

Evaluating reward models usually starts with accuracy on held-out preferences. Give the reward model a prompt, a chosen response and a rejected one, and count how often it scores the chosen one higher. The Allen Institute for AI's RewardBench turned that into a shared benchmark. Its successor, RewardBench 2, made the test harder by pairing each correct answer with three wrong ones. Models scored about 20 points lower on it.

The RewardBench 2 authors limit what accuracy proves: strong scores are "a prerequisite for strong training with RLHF, but they are not sufficient." Accuracy tracked the quality of best-of-N sampling (generating several answers and keeping the top-scored one) well. During RL training with PPO, though, results levelled off once a reward model was reasonably good.

Two other papers push further:

  • A Princeton team proved in What Makes a Reward Model a Good Teacher? that training stalls if a reward model gives similar scores to everything the policy produces, however accurate the model is. In their words, "even a perfectly accurate reward model can lead to extremely slow optimization."
  • An ICLR 2025 study, Rethinking Reward Model Evaluation, measured the link directly. It found "a weak positive correlation between accuracy and downstream performance." Reward models with similar accuracy produced policies of different quality.

ReVeR's GRPO result from the introduction is the same finding on rubric reward models. The authors conclude that "higher preference accuracy does not necessarily translate into better downstream policies." A reward model has to be judged by what it teaches, and accuracy is only the first filter.

In ReVeR's GRPO test, the more accurate reward model (66.3%) left the policy below its 53.9 starting score at 51.8, while the less accurate one (63.9%) raised it to 55.1.

What Are Rubrics as Rewards?

A rubric reward replaces "which answer is better?" with "which of these criteria does this answer meet?". A judge scores the response against a written list of criteria, and that score becomes the reward. The approach, known as rubrics as rewards, extends reinforcement learning to answers that no program can check. That makes it a candidate for aligning large language models in expert domains.

Published designs differ in where the criteria come from. Checklists Are Better Than Reward Models, presented at NeurIPS 2025, extracts a checklist from each instruction and scores every item with AI judges and specialised verifier programs. OpenRubrics generates rubrics by contrasting preferred and rejected responses, then discards rubrics that contradict the human preference labels.

Rubric rewards have two advantages over a single preference score. They are readable: an expert can see which criterion failed. They can also be absolute. NVIDIA's HelpSteer2-Preference team points out that "Bradley-Terry Models are not calibrated": a Bradley-Terry score only means something relative to another response. A rubric score says whether an answer met a standard.

A preference reward asks which of two answers is better and produces a relative score; a rubric reward asks which written criteria an answer meets and produces an absolute score.

ReVeR adds a design rule. Its authors note that existing methods "often derive criteria from observed response candidates", which lets the responses shape the standard. ReVeR writes rubrics from the instruction alone and splits them into hard rules and graded principles. It then scores each response separately. Tools check the mechanical constraints that a reading-only judge struggles with.

The weak point moves with it. The reward is now only as good as the rubric, and in both OpenRubrics and ReVeR, a model writes the rubric. RubricBench, an ACL 2026 benchmark, gave the same judges human-written and model-written rubrics. Human rubrics won by 22.1 to 28.3 points. One judge scored 38.8% with no rubric, 57.8% with rubrics it wrote itself and 84.9% with human rubrics.

How Do Rubric Rewards Work in Law, Medicine and Insurance?

These are the domains where a preference label is hardest to get from a generalist, and where a written standard is easiest to defend. The evidence is uneven across the three.

Law

LegalRewardBench describes itself as the first human-verified benchmark for reward models in law. It holds 1,331 preference pairs across five levels: knowledge, understanding, analysis, decision and writing. Legal annotators wrote rubrics from reference answers, an LLM judge scored candidate responses against them, and five law master's students then verified the pairs over 560 human-hours. A double-blind check on 5% of pairs found about 80% agreement.

The results favour general models. The top general-purpose judge scored 86.85% overall, and the best fine-tuned reward model 78.25%. The two best fine-tuned models scored lowest on understanding, then writing; the authors write that current models "still struggle on information extraction and generation tasks". The tasks are mainly Chinese, so the scores may not carry over to other legal systems.

Another legal benchmark shows why the human reference matters. LEXam is an ICLR 2026 benchmark built from 340 law exams. Three legal experts rated the same 50 answers, with a mean quadratic weighted κ of 0.49 between them. κ is an agreement score corrected for chance, where 1 means perfect agreement. The gaps between the LLM judge and the humans were largest on the answers where the experts disagreed most.

Medicine

OpenAI's HealthBench is the best-known rubric set in medicine: 5,000 conversations, with criteria written by 262 physicians. "Disagreement was inherent in the exercise," the authors write, and both physician-to-physician and model-to-physician agreement ranged from 55% to 75%. Apart from the consensus criteria, the conversation-specific criteria were written by individual physicians and "not validated by other physicians".

Not every expert-written rubric holds up. In a Canvas Medical study, clinicians created 5,797 rubrics for clinical cases, and 3,060 passed validation. Judges that apply the criteria have their own gap. A 2026 preprint, Same Verdict, Different Reasons, found that LLM judges spotted incomplete medical answers "at and slightly above near-chance levels". It also found that "even when model and clinician verdicts agree, they rarely cite the same explanation."

Insurance

We found no published reward model or rubric reward study for insurance. The nearest evidence is evaluation. INSEva is a Chinese insurance benchmark from Ant Group and university partners. It grades open answers with an LLM. Checked against human ratings, the paper reports 73.60 for faithfulness but 55.78 for completeness, which tests whether an answer covers every point in the reference. Regulators already ask about the human side: the US NAIC model bulletin on insurers' use of artificial intelligence asks them to weigh "the extent to which humans are involved in the final decision-making process."

How Do Reward Models Fail Before Anyone Games Them?

"How they fail" usually means reward hacking: the policy finds a way to score highly without doing the task. Our RLVR guide covers that failure in detail. The failures below come earlier. They are problems with the reward model as a measurement, and they exist before any policy is trained against it.

Disagreement becomes confidence

Diverging Preferences, an ICML 2025 study, found that diverging preferences made up over 30% of examples in two preference datasets. Over 75% of the disagreements involved response complexity, verbosity or underspecified prompts. Bradley-Terry reward models predicted reward differences on these contested pairs at a rate "akin to that of high-agreement preferences". The model reports certainty that the people never had.

Scores are relative, and judges have order effects

A Bradley-Terry reward model ranks one response against another; it says nothing about whether either is acceptable. LLM judges used as pairwise reward models add position effects. In the MT-Bench study, GPT-4 gave the same verdict when the two answers were swapped 65.0% of the time, and Claude-v1 23.8% of the time.

The domain shifts

Reward models lose accuracy on prompts unlike their training data. An EMNLP 2024 study of implicit reward models found that both standard reward models and DPO's built-in one dropped out of distribution on every base model tested. One standard reward model fell from 75.3% to 62.2%. LegalRewardBench shows the same pattern in a professional domain. The two best fine-tuned reward models trailed a general judge on all five levels. In the authors' words, they "transfer less reliably to legal preference evaluation".

The rubric drifts from the experts

ReVeR also measured how closely its generated rubrics matched expert-written ones. Further training of the rubric writer raised preference accuracy and lowered that match. Agreement with expert rubrics, measured as F1 (an overlap score), fell from 32.1 to 27.0 on chat prompts and from 46.3 to 38.1 on instruction-following prompts. The rubrics got better at predicting labels and less like what an expert would write.

The judge agrees for the wrong reasons

LEXam and Same Verdict, Different Reasons point at the same gap. An LLM judge can match an expert's verdict while relying on different evidence, and it diverges most where experts themselves disagree. Agreement on the final score can hide disagreement on why.

How Should Domain Experts Build and Check a Reward Model?

Each failure above has a check that only someone with domain knowledge can run.

  1. Measure the experts before the model. Have several experts label the same items and record their agreement. That figure is the ceiling the reward model is measured against. Our Who Judges the Judge? data story sets out a validation protocol: a gold set of 50 to 100 items, at least 75% agreement, read alongside a chance-corrected figure.
  2. Drop or adjudicate contested items. Diverging Preferences shows what happens if you average them in. Keep each expert's label so the disagreement stays visible.
  3. Write criteria from the task, not the answers. Follow ReVeR's rule: derive the rubric from the instruction, then score responses against it.
  4. Have experts review machine-written rubrics. RubricBench's gap and ReVeR's fidelity drop both say a generated rubric needs a domain check. Track how many criteria an expert edits or removes.
  5. Keep a gold set of known answers. A golden dataset scored by experts lets you test both the reward model and each expert against the same reference.
  6. Test in the target domain, then test the policy. An in-domain held-out set catches distribution shift. A downstream check on the trained model catches the accuracy-is-not-teaching problem.

Where Kili fits

Kili is where the domain expert does this judging and where the platform measures whether that judgment holds. You import the response as a text or PDF asset. Each rubric criterion becomes a single-choice classification job (met or not met), so the expert answers every criterion separately. With consensus enabled on the labelling step, several experts judge the same asset blind. That tests whether a criterion is clear enough for experts to agree on. A honeypot, enabled on one workflow step, scores each expert's labels on that step against known answers. Python plugins run on Kili when an expert submits or reviews an asset. A plugin can check a business rule (every criterion answered, say), add an issue and send the asset back to the labelling queue.

A model response is scored against rubric criteria as met or not met, with consensus checking expert agreement and honeypot checking against known answers; a plugin checks a business rule and sends failures back before the data reaches your training stack.

For side-by-side comparison of model responses, Kili's LLM labeling projects let experts rank and rate responses. That interface is in private beta; ask the Kili team for access. Reward model training itself runs in your own stack.

Conclusion: Reward Models Are a Record of Who Judged

The 13 October papers point to the same shift. Reward models are moving from opaque preference scores to written rubrics, and toward judges that read the criteria and check the evidence. That makes the reward easier to inspect. It also means the rubric, and the people who wrote and checked it, now carry most of the weight.

Accuracy on a preference benchmark will not catch the failures that matter in law, medicine and insurance. A reward model can be accurate and teach badly, accurate and drift from expert rubrics, or accurate on average and wrong on exactly the cases experts dispute. Teams building reward models for expert domains should report three numbers: agreement among their experts, the reward model's accuracy against those experts, and the change in the trained policy. ReVeR's authors reached the same conclusion from the model side, treating rubric fidelity, accuracy and downstream use as separate properties. The first of those numbers can only come from the experts.

Resources

Peg papers (October 2026)

Reward model foundations

Evaluating reward models

Failure modes

Rubric rewards

Law

Medicine

Insurance

From Kili Technology

Ready to Put Experts Behind Your Reward Model?

Talk to the Kili team about expert rubric review with consensus and honeypot checks configured in each project's workflow. The pairwise ranking interface is in private beta; ask us for access.

Related articles

Reward Models Explained: Preference Data, Rubric Rewards and How They Fail
November 9, 2026

Reward Models Explained: Preference Data, Rubric Rewards and How They Fail

A reward model turns human judgment into a training score. Learn how preference and rubric reward models work in law, medicine and insurance, and how they fail.

Kili Technology
Kili Technology
LLMs
AI Evaluation
Fine-Tuning Audio Models Guide: What Annotation Quality Decides [2026]
September 3, 2026

Fine-Tuning Audio Models Guide: What Annotation Quality Decides [2026]

Fine-tuning audio models — ASR, diarization, speech-to-text — is bounded by your reference transcripts. What to annotate, in what order, and how to avoid circular training data.

Kili Technology
Kili Technology
Audio
Data Labeling
Speaker Diarization Models Guide: Benchmarks and Failure Modes [2026]
August 7, 2026

Speaker Diarization Models Guide: Benchmarks and Failure Modes [2026]

Speaker diarization looks solved on benchmark corpora and clearly is not on real recorded calls. This guide explains what diarization error rate actually measures, why the figures aren't comparable across sources, and which failure modes benchmarks exclude.

Kili Technology
Kili Technology
AI Evaluation
Audio
What's next?

Ready when you are. Start your free trial.

Start free trial
Share Your Use Case

Kili Technology is the complete platform to build trustworthy, high-quality datasets for training, fine-tuning, and evaluating AI/ML models.

Product
Deploy Your Way
Auditable from Day Zero
Quality-First Design
Expert-in-the-Loop
Company
© 2025 Kili Technology. All rights reserved.
Privacy Policy
Legal Notice
SecurityInfo