DPO vs RLHF Explained: What Changes When You Drop the Reward Model

Introduction

Direct Preference Optimization (DPO) has become the default way to align open large language models. Its appeal is simplicity: it replaces most of the pipeline behind reinforcement learning from human feedback (RLHF). Both methods align a language model with human preferences, and both learn from the same kind of human preference data. RLHF trains a separate reward model on that data. It then optimises the language model against that model's scores with reinforcement learning (RL), where a model improves by trial and error toward a reward. DPO skips both steps and trains the model on the preference pairs directly.

The comparison most teams run is DPO vs PPO (Proximal Policy Optimization), the RL algorithm in classic RLHF. A 2024 study from AI2 and the University of Washington put that choice in proportion. Ivison et al. (NeurIPS 2024) tested four factors in preference training. Switching from DPO to a well-tuned PPO improved results by up to 2.5%. Switching to better preference data improved them by up to 8%. The authors found "better preference data leading to the largest improvements."

DPO removes the reward model. It does not remove the preference data, or the people who ranked it. With no reward model in between, each pair's label becomes the training signal, so every quality problem moves upstream onto the pairs.

This practical guide explains how DPO and RLHF work and how they compare. It then covers three questions that decide whether DPO works in practice. What does a usable preference pair look like? When does pairwise comparison beat rubric scoring? And what level of agreement makes a label safe to train on?

What Is Direct Preference Optimization (DPO)?

Direct Preference Optimization is a method for aligning a language model with human preferences by training it on pairs of responses, one preferred and one rejected, without a separate reward model or reinforcement learning. The model learns to generate text people prefer. It was introduced by Rafailov et al. at NeurIPS 2023, where it was runner-up for the Outstanding Paper award.

The training dataset is a set of preference pairs: two different responses to the same prompt, labelled by human annotators or by another model. Each pair holds the prompt, the preferred response ("chosen") and the dispreferred one ("rejected"). Hugging Face's TRL library gives the simplest example: prompt "The sky is", chosen " blue.", rejected " green."

Here is the DPO loss function in one sentence. For each pair, it raises the probability of the chosen response and lowers the probability of the rejected one, both relative to a frozen reference model. The reference is usually the initial model you started from: a base model after supervised fine-tuning (SFT). Calculating DPO loss therefore needs the probability of both responses under two models, the policy and the reference. A parameter, β, controls how far the trained model may drift from that reference. TRL's documentation puts it simply: "Higher β means less deviation from the reference model."

Mathematically, DPO targets the same RLHF objective: maximise reward while staying close to the reference. But it rewrites the reward in terms of the policy (the model being trained) itself. Hence the subtitle "Your Language Model Is Secretly a Reward Model." Directly optimizing the policy on the pairs turns RLHF's two-stage process into a single classification loss. There is no reward model training, no sampling from the model and no RL loop. The authors describe the result as "stable, performant, and computationally lightweight."

How Does RLHF Use a Reward Model?

Reinforcement learning from human feedback (RLHF) was the first widely used method for incorporating human feedback into language generation. It first learns a reward function from human preferences, then optimises the model against it. Christiano et al. (NeurIPS 2017) introduced the approach for RL agents. Their agents learned complex tasks with human feedback on less than 1% of their interactions. A small amount of comparison data goes a long way. Then Stiennon et al. (NeurIPS 2020) applied it to summarisation. InstructGPT (Ouyang et al., NeurIPS 2022) made the three-step recipe standard:

1. Supervised fine-tuning on human-written demonstrations.
2. Reward model training on ranking data. Labellers ranked between four and nine model responses per prompt. The reward model learns to predict which response a person would prefer.
3. Reinforcement learning with PPO. The policy function, meaning the model's mapping from prompts to responses, is updated to maximize rewards from the reward model. A penalty keeps it close to the SFT model.

The result was a 1.3-billion-parameter InstructGPT model whose outputs people preferred to those of the 175-billion-parameter GPT-3.

Both RLHF and DPO rest on the same preference model, the Bradley-Terry model. It treats the chance that one response beats another as a function of the gap between their hidden quality scores. RLHF fits those scores with a separate network; DPO fits them inside the policy.

What RLHF costs

RLHF requires multiple models during training. PPO is an actor-critic algorithm. Alongside the policy (the actor), it trains a value model (the critic) that estimates how good each state is. InstructGPT initialised that value model from the reward model. Add the frozen reference and the reward model itself, and four large models are in play. ReMax (Li et al., ICML 2024) describes PPO as leading to "laborious hyper-parameter tuning and significant computation burdens." Removing the value model alone saved about 46% of GPU memory on a 7B model. The DPO paper puts it plainly: "RLHF is a complex and often unstable procedure."

PPO also has something DPO lacks. It samples new responses from the current model during the training process, scores them with the reward model, and reinforces those with higher rewards. RLHF can learn from outputs that were never in the original dataset. Some teams also collect human feedback in rounds. Anthropic updated its preference models and RLHF policies "on a weekly cadence with fresh human feedback data."

The reward signal has a known weakness: it is a proxy for human judgement. Gao, Schulman and Hilton (ICML 2023) found that optimising too hard against it can "hinder ground truth performance." Kili's guide to RLVR covers this reward hacking in more detail.

The reward model also does useful work. The reward learning step produces a separate artefact you can test and fix before any policy training. It also averages noisy preferences into a score. DPO removes that stage.

DPO vs RLHF: What Changes When You Drop the Reward Model?

DPO simplifies the RLHF process. It never fits an explicit reward function, it needs fewer models and less tuning, and it trains on a static dataset collected in advance. The trade-off is that the policy now learns from each preference label directly.

What DPO keeps from RLHF

Four things stay.

DPO still needs a reference model, and still constrains the policy to stay near it. Only later variants such as SimPO and ORPO remove the reference.

DPO still overoptimises, a form of reward hacking against its implicit reward. A team that included DPO's own authors, Rafailov et al. (NeurIPS 2024), studied direct alignment methods like DPO. They found these methods "still commonly deteriorate from over-optimization," often before one pass through the dataset.

DPO still assumes Bradley-Terry. Azar et al. (AISTATS 2024) showed what happens when one response wins almost every time: DPO's regularisation weakens and the model overfits. Their fix became the identity preference optimization variant, IPO.

DPO can still learn the wrong lesson from length. Park et al. (Findings of ACL 2024) found DPO exploits verbosity. Controlling for length improved length-controlled win rates (how often a judge prefers the model's answer, adjusted for length) by up to 20%.

Frontier pipelines haven't dropped reward models either. Meta's Llama 3 report uses one to pick the best of several sampled responses, then aligns the policy with DPO.

DPO vs PPO: Which Performs Better?

Neither wins outright. Results vary depending on the task, the tuning budget and, most of all, the data.

The original DPO paper reported that DPO matched or improved on PPO-based RLHF in summarisation and single-turn dialogue, and exceeded it on sentiment control. Later controlled studies favour PPO. Xu et al. (ICML 2024) tested both on tasks from dialogue to code generation. They found "PPO is able to surpass other alignment methods in all cases." Ivison et al. also found PPO ahead, by up to 2.5% on maths and 1.2% in general.

The case for DPO comes from cost and simplicity at scale. Meta reports that, for Llama 3, it "found that DPO required less compute for large-scale models and performed better" on instruction following. AI2's Tülu 3 recipe uses a length-normalised DPO between SFT and RL. Hugging Face's Zephyr showed strong performance from DPO on preferences from another model entirely. GPT-4 scored the responses, and the highest-scoring one became "chosen". Collecting human feedback is slow and costly. Google researchers (Lee et al., 2023) note that "gathering high-quality preference labels is expensive." They found AI feedback performed comparably to human feedback on the tasks they tested. Note that Llama 3, Tülu 3 and Zephyr are each a lab's report on its own model.

In practice, DPO suits teams with limited compute and a reliable preference dataset. PPO is worth its cost in three cases: you can afford the tuning, you have a reward model you trust, and the model needs to keep learning from fresh samples on complex tasks such as maths and code.

The practical question is often which signal you have. Pairs suit DPO. A single thumbs-up or thumbs-down per output suits KTO, which needs no pairs. A verifiable answer suits RL with a rule-based reward. Comparing several sampled answers to the same prompt is the idea behind GRPO, covered in our GRPO blog post. Kili's 2026 RLHF guide maps where each method sits in the post-training pipeline.

Why Does Preference Data Quality Decide How Well DPO Works?

Because each pair is a training example for the policy itself. If the label is wrong, DPO pushes the model toward the worse response.

Preference labels are noisy. InstructGPT's labellers agreed with each other about 73% of the time. Anthropic's HH-RLHF paper (Bai et al., 2022) "found poor average agreement (about 63%) between Anthropic researchers and our crowdworkers." A Fudan University team (Wang et al., 2024) scored the HH-RLHF pairs with ten reward models. They estimated that about 25% likely had the preference the wrong way round.

That noise has a measurable cost in DPO. Gao, Alon and Metzler (Google Research, 2024) trained DPO models on preference data with controlled amounts of noise. Each additional ten percentage points of noise could lower win rate by up to 30 points. Win rate here is how often a judge prefers the model's response over a baseline's. Ray Chowdhury et al. (ICML 2024) found "the performance of DPO drops significantly when the noise rates are high." Their corrected loss function needs an estimate of the noise rate.

RLHF is not immune; its reward model learns from the same noisy pairs. The difference is where you can catch the problem. In RLHF, the reward model is a separate checkpoint you can evaluate against held-out expert judgements before the policy sees it. In DPO, the first artefact trained on the pairs is the model you intend to ship. The dataset is the only checkpoint.

Who ranked the pairs matters. Sharma et al. (ICLR 2024) found that humans and preference models sometimes "prefer convincingly-written sycophantic responses over correct ones." Hosking, Blunsom and Bartolo (ICLR 2024) found preference scores "under-represent important aspects like factuality." Assertive answers also made annotators perceive fewer errors. A ranker who can't check the facts rewards the answer that sounds right. DPO trains that preference straight into the weights. Preference data stands in for human values only as well as the people who produced it.

Neither method guarantees factual accuracy, fairness or safety on its own. InstructGPT improved truthfulness and reduced toxic outputs. But it "does not significantly improve over GPT-3" on the Winogender bias benchmark. The authors also note it "still makes simple mistakes."

Alignment can also cost some natural language understanding. InstructGPT showed "performance regressions compared to GPT-3 on certain public NLP datasets, notably SQuAD," a reading-comprehension benchmark. OpenAI called this an alignment tax and reduced it by mixing pretraining updates into PPO.

What Does a Usable Preference Pair Look Like?

A usable pair has a clear winner, a winner that is actually good, and a difference an expert could explain. Five checks follow from the research.

The margin is clear

Similar responses produce unreliable labels. Meta's Llama 2 report asked human annotators to grade each preference as significantly better, better, slightly better, or negligibly better/unsure. Both annotator agreement and reward model accuracy dropped as the two responses became more similar. The authors concluded "the accuracy on more distinct responses matters the most." Collect a strength rating with every preference, then treat "negligibly better" pairs as candidates to drop.

The chosen response is good, not just better

A pair where both responses are wrong still teaches the model to prefer one of them. If the preferred response contains an error, DPO reinforces the error. Ask the reviewer to flag "both unacceptable" as a separate outcome. Send those prompts back for new responses instead of training on them.

Length and tone aren't doing the deciding

Singhal et al. (2024) found that a reward based on length alone nearly matched standard RLHF on one benchmark, because longer answers kept winning. If chosen responses are systematically longer, more confident or more polite than rejected ones, the model learns that pattern. Check the length gap across your training dataset before you start.

The responses come from the model you are training

Pairs built from another model's outputs teach the policy about text it would never write. Tajwar et al. (ICML 2024) found that methods using "on-policy sampling" from the model being trained outperformed purely offline ones. Generate the candidate responses with the current model, or a close version of it, where you can.

Fewer, cleaner pairs beat more pairs

Deng et al. (NeurIPS 2025) selected DPO pairs by margin and trained on 10% of a standard dataset. They reported gains of 3% to 8% over training on all of it. Their summary: "More data in DPO training does not always yield better results."

When Does Pairwise Comparison Beat Rubric Scoring?

Pairwise comparison works best when quality is a matter of judgement and hard to write down. Rubric scoring works best when an expert can state the criteria in advance.

The evidence for pairwise comes mostly from human annotation research. Kiritchenko and Mohammad (ACL 2017) found a comparative method "produces significantly more reliable results than the rating scale" at the same budget. Novikova et al. (NAACL 2018) found relative assessment made human ratings for text generation more consistent. People find it easier to pick the better of two answers than to score one out of 7.

The evidence for rubrics comes from tasks with checkable criteria. DeepMind's work on Sparrow (Glaese et al., 2022) studied the alignment of dialogue agents. The team broke good dialogue into natural-language rules to "collect more targeted human judgements" alongside pairwise comparisons. Rubrics as Rewards (Gunjal et al., ICLR 2026) is by Scale AI researchers. It found rubrics written for each prompt outperformed direct Likert judging (a single score on a fixed scale) by up to 31% on the HealthBench medical benchmark. Viswanathan et al. (NeurIPS 2025) found checklist feedback "is the only method to help on every benchmark" they tested for instruction following. Pairwise has its own failure mode too. Tripathi et al. (2025) tested LLM judges, meaning language models used to grade outputs. Distracting features flipped their pairwise preferences in about 35% of cases, against 9% for absolute scores.

The collection format may matter less than either camp assumes. NVIDIA's HelpSteer2-Preference (Wang et al., ICLR 2025) labelled the same responses both ways, as Likert ratings and as pairwise preferences. Reward models trained on each performed comparably.

A practical rule follows:

• Use pairwise preference when the criteria are subjective (tone, helpfulness, style) and you plan to train with DPO.
• Use a rubric when the expert can write the standard down, such as required clinical facts or regulatory clauses. A rubric also tells you why an answer lost.
• Combine them: ask for a pairwise choice plus a short rubric check on the chosen response. That catches "both bad" pairs and gives you a reason to filter on.

Choose the label format before you choose the algorithm, since the format determines which algorithms you can use.

What Agreement Threshold Makes a Preference Label Safe to Train On?

There is no universal number. Published pipelines each set their own threshold and measure agreement differently. What they share is a rule: disagreement removes a pair from training, or sends it to adjudication.

Three examples show the range:

• HelpSteer2-Preference sent each comparison to three to five annotators on a seven-point scale. It kept the three closest ratings and dropped the item if they spread by more than two points. Items averaging zero were also dropped. Raw agreement was 0.489 on a weighted Cohen's kappa (κ), an agreement score corrected for chance where 1 is perfect. The authors called it "moderate." Dropping the outliers (10% of tasks) raised it to 0.843. Dropping the items that averaged zero (another 22%) raised it to 0.878. Reliable labels cost about a third of the data.
• Gao et al. kept only pairs whose preference confidence was above 0.8. That consistently improved DPO results and removed over half the data. Pushing the threshold to 0.99 hurt.
• Anthropic's HH-RLHF dropped comparisons where the worker chose the weakest preference option.

The headline numbers in this field aren't comparable with each other. InstructGPT's 73% is raw percent agreement between labellers. HelpSteer2's 0.843 is a chance-corrected kappa after filtering. Ties alone move the number. On MT-Bench, a set of multi-turn chat questions, Zheng et al. (NeurIPS 2023) measured human-to-human agreement of 63% with ties counted and 81% without. Our data story Who Judges the Judge? shows how far raw agreement can overstate reliability once chance agreement is removed.

A workable data collection policy for a DPO dataset:

1. Calibrate on a gold set first. Before ranking at scale, have experts label 50 to 100 pairs with known answers. Kili's guide to golden datasets covers how to build one. Start production ranking when reviewers reach at least 75% agreement with it. Read that figure alongside a chance-corrected one such as Cohen's κ.
2. Overlap a share of production pairs. While collecting data, route a sample to two or more experts. Track agreement by task type, since some slices will be much harder than others.
3. Drop or adjudicate disagreements. A pair where qualified experts split is not a reliable training signal. Either remove it or send it through an approval process where a senior reviewer decides, and record that decision.
4. Drop weak preferences. Use the strength rating to exclude "negligibly better" pairs, as Llama 2 and HH-RLHF did.
5. Watch the agreement trend. Falling agreement on a slice usually means the guideline is unclear, not that the experts got worse.

Where Kili fits

Kili doesn't run DPO. The training loop runs in your own stack. Kili is where domain experts rank and check the preference pairs before they become training data. Each prompt and its two candidate responses are imported as a text asset. The expert records which response is better, and by how much, in a single-choice classification job (for example, from "A much better" to "B much better"). Consensus, enabled on a labelling step, measures how often experts agree when they label the same pair, without seeing each other's answers. Honeypot scores each expert's work against pairs with known answers, which is the gold-set check in step 1. Enforce Step Separation, on by default, means no one reviews their own work. Kili's LLM labeling projects, designed for pairwise comparison of model responses, are in private beta; ask the Kili team for access.

Conclusion: DPO Makes the Dataset the Model's Only Teacher

RLHF had two teachers: the human rankers, and the reward model that summarised them. The reward model was imperfect and could be gamed, but it was also a place to look. You could test it against your experts and fix it before the policy learned anything.

DPO removes that second teacher. What's left is the human preference data, and the policy learns from every label in it, right or wrong. That moves the engineering effort from machine learning infrastructure to data. In an artificial intelligence system trained this way, the learning process is only as good as the labels. The work that matters most in a DPO project happens before training starts. Decide the label format. Choose who ranks. Measure whether they agree, and drop the pairs where they don't. Teams that treat preference pairs as curated, expert-checked, high-quality preference data get the simplicity DPO promised. Teams that treat them as bulk input will be debugging the model when the problem was in the data.

Resources

DPO and RLHF foundations

• Direct Preference Optimization: Your Language Model Is Secretly a Reward Model – Rafailov et al. (2023), NeurIPS
  • https://arxiv.org/abs/2305.18290
• Deep Reinforcement Learning from Human Preferences – Christiano et al. (2017), NeurIPS
  • https://arxiv.org/abs/1706.03741
• Learning to Summarize from Human Feedback – Stiennon et al. (2020), NeurIPS
  • https://arxiv.org/abs/2009.01325
• Training Language Models to Follow Instructions with Human Feedback (InstructGPT) – Ouyang et al. (2022), NeurIPS
  • https://arxiv.org/abs/2203.02155
• DPO Trainer documentation – Hugging Face TRL
  • https://huggingface.co/docs/trl/dpo_trainer

DPO vs PPO and DPO variants

• Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback – Ivison et al. (2024), NeurIPS
  • https://arxiv.org/abs/2406.09279
• Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study – Xu et al. (2024), ICML
  • https://proceedings.mlr.press/v235/xu24h.html
• The Llama 3 Herd of Models – Grattafiori et al. (2024), arXiv, Meta
  • https://arxiv.org/abs/2407.21783
• Tülu 3: Pushing Frontiers in Open Language Model Post-Training – Lambert et al. (2024), arXiv, AI2
  • https://arxiv.org/abs/2411.15124
• Zephyr: Direct Distillation of LM Alignment – Tunstall et al. (2023), arXiv, Hugging Face
  • https://arxiv.org/abs/2310.16944
• ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models – Li et al. (2024), ICML
  • https://proceedings.mlr.press/v235/li24cd.html
• RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback – Lee et al. (2023), arXiv, Google
  • https://arxiv.org/abs/2309.00267
• Model Alignment as Prospect Theoretic Optimization (KTO) – Ethayarajh et al. (2024), ICML
  • https://proceedings.mlr.press/v235/ethayarajh24a.html
• ORPO: Monolithic Preference Optimization without Reference Model – Hong, Lee & Thorne (2024), EMNLP
  • https://aclanthology.org/2024.emnlp-main.626
• SimPO: Simple Preference Optimization with a Reference-Free Reward – Meng, Xia & Chen (2024), NeurIPS
  • https://proceedings.neurips.cc/paper_files/paper/2024/hash/e099c1c9699814af0be873a175361713-Abstract.html

Overoptimisation and failure modes

• Scaling Laws for Reward Model Overoptimization – Gao, Schulman & Hilton (2023), ICML
  • https://proceedings.mlr.press/v202/gao23h.html
• Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms – Rafailov et al. (2024), NeurIPS
  • https://proceedings.neurips.cc/paper_files/paper/2024/hash/e45caa3d5273d105b8d045e748636957-Abstract.html
• A General Theoretical Paradigm to Understand Learning from Human Preferences (IPO) – Gheshlaghi Azar et al. (2024), AISTATS
  • https://proceedings.mlr.press/v238/gheshlaghi-azar24a.html
• Disentangling Length from Quality in Direct Preference Optimization – Park et al. (2024), Findings of ACL
  • https://aclanthology.org/2024.findings-acl.297/
• A Long Way to Go: Investigating Length Correlations in RLHF – Singhal et al. (2024), arXiv
  • https://arxiv.org/abs/2310.03716

Preference data quality and noise

• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback – Bai et al. (2022), arXiv, Anthropic
  • https://arxiv.org/abs/2204.05862
• Llama 2: Open Foundation and Fine-Tuned Chat Models – Touvron et al. (2023), arXiv, Meta
  • https://arxiv.org/abs/2307.09288
• Secrets of RLHF in Large Language Models Part II: Reward Modeling – Wang et al. (2024), arXiv technical report
  • https://arxiv.org/abs/2401.06080
• Impact of Preference Noise on the Alignment Performance of Generative Language Models – Gao, Alon & Metzler (2024), arXiv, Google Research
  • https://arxiv.org/abs/2404.09824
• Provably Robust DPO: Aligning Language Models with Noisy Feedback – Ray Chowdhury, Kini & Natarajan (2024), ICML
  • https://proceedings.mlr.press/v235/ray-chowdhury24a.html
• Less is More: Improving LLM Alignment via Preference Data Selection – Deng et al. (2025), NeurIPS
  • https://arxiv.org/abs/2502.14560
• Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data – Tajwar et al. (2024), ICML
  • https://proceedings.mlr.press/v235/tajwar24a.html
• Towards Understanding Sycophancy in Language Models – Sharma et al. (2024), ICLR
  • https://arxiv.org/abs/2310.13548
• Human Feedback Is Not Gold Standard – Hosking, Blunsom & Bartolo (2024), ICLR
  • https://arxiv.org/abs/2309.16349

Pairwise vs rubric scoring and agreement

• HelpSteer2-Preference: Complementing Ratings with Preferences – Wang et al. (2025), ICLR, NVIDIA
  • https://arxiv.org/abs/2410.01257
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena – Zheng et al. (2023), NeurIPS Datasets and Benchmarks
  • https://arxiv.org/abs/2306.05685
• Best-Worst Scaling More Reliable than Rating Scales – Kiritchenko & Mohammad (2017), ACL
  • https://aclanthology.org/P17-2074/
• RankME: Reliable Human Ratings for Natural Language Generation – Novikova, Dušek & Rieser (2018), NAACL
  • https://aclanthology.org/N18-2012/
• Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation – Tripathi et al. (2025), arXiv
  • https://arxiv.org/abs/2504.14716
• Improving Alignment of Dialogue Agents via Targeted Human Judgements (Sparrow) – Glaese et al. (2022), arXiv, DeepMind
  • https://arxiv.org/abs/2209.14375
• Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains – Gunjal et al. (2026), ICLR, Scale AI
  • https://arxiv.org/abs/2507.17746
• Checklists Are Better Than Reward Models for Aligning Language Models – Viswanathan et al. (2025), NeurIPS
  • https://proceedings.neurips.cc/paper_files/paper/2025/hash/a6837c1dd021f76f1b4098e3722052a8-Abstract-Conference.html

Kili guides

• 2026 RLHF Guide – where SFT, preference optimisation and RL fit in post-training
  • https://kili-technology.com/blog/exploring-reinforcement-learning-from-human-feedback-rlhf-a-comprehensive-guide
• Who Judges the Judge? – raw agreement vs chance-corrected agreement, and expert vs crowd labels
  • https://kili-technology.com/blog/data-story-who-judges-the-judge-llm-as-a-judge
• How to Build Golden Datasets for Testing, Fine-Tuning and Evaluating AI Models
  • https://kili-technology.com/blog/how-to-build-golden-datasets-for-testing-fine-tuning-and-evaluating-ai-models
• RLVR Explained: Reinforcement Learning with Verifiable Rewards – how reward signals get gamed
  • https://kili-technology.com/blog/rlvr-reinforcement-learning-with-verifiable-rewards
• GRPO Explained: Group Relative Policy Optimization – Kili blog post
  • https://kili-technology.com/blog/grpo-group-relative-policy-optimization
• Talk to the Kili team
  • https://kili-technology.com/contact