.png)
_logo%201.svg)
AI Summary
- SFT, preference optimisation and RLVR ask humans to write, compare and specify, in that order.
- InstructGPT's own labelers agreed on only 72.6% of comparisons.
- DPO removed PPO from the pipeline, not the human preference data.
- Verifiable rewards move expert judgment into the verifier, which models learn to game.
Introduction
When this guide was first published in 2024, reinforcement learning from human feedback (RLHF) meant one recipe: fine-tune a model on human-written examples, train a reward model on human rankings, then optimise the model against that reward with PPO (proximal policy optimisation, a reinforcement learning algorithm). RLHF was the name for the whole of alignment training.
That is no longer how the labs describe it. Meta's Llama 3 report defines post-training as any model training that happens outside of pre-training, and runs several rounds of it. AI2's Tülu 3 names three stages outright: supervised fine-tuning, preference optimisation, and reinforcement learning with verifiable rewards (RLVR). Nathan Lambert's RLHF Book now places RLHF inside the second stage, as one method among several.
Most explainers organise this pipeline by algorithm: PPO, DPO, GRPO and the rest. This one organises it by the question a machine learning team actually has to answer when it staffs a project. What is the human being asked to do at each stage, and how would you know if they did it well? The answer changes from stage to stage, and so does the way things go wrong.
What Is RLHF, and Why Is It No Longer the Whole Story?
Large language models get their broad language ability in pre-training, where self-supervised learning on massive text datasets teaches them to predict the next token. That produces a capable base model, but not a helpful one. Reinforcement learning from human feedback (RLHF) closes the gap: human preferences become a reward signal, and reinforcement learning tunes the model to score well against it. Collecting human feedback costs far more per example than pre-training on raw text, so RLHF runs on much smaller datasets and scales less easily.
Learning from human feedback started outside language models. In 2017, researchers at OpenAI and DeepMind trained agents on people's comparisons of short video clips. The agents learned complex behaviours from feedback on less than 1% of their interactions, and some tasks took about an hour of human time. Three years later, OpenAI applied the same loop to summarisation. Humans compared summaries, a reward model learned their preferences, and reinforcement learning optimised against it. The resulting summaries outperformed both human-written reference summaries and much larger models.
InstructGPT made the recipe famous in 2022. Its headline result still holds: outputs from a 1.3-billion-parameter InstructGPT model were preferred to outputs from 175B GPT-3, a model with 100 times more parameters. The comparison is with the base GPT-3, and the difference came from the post-training data.
OpenAI did not simply hire labelers; it screened about 40 contractors with a test and selected on performance. And the authors say plainly that the model is aligned to those labelers' preferences, not to human values in general. RLHF was always a bet on specific people's judgment.
The paper also measured side effects. InstructGPT generated about 25% fewer toxic outputs than GPT-3 when prompted to be respectful, and made up information about half as often on closed-domain tasks (21% against 41%). It did not significantly improve on the Winogender and CrowS-Pairs bias benchmarks. RLHF reduced some risks and left others where they were.
What changed since then is the shape of the pipeline around that judgment. The classic four steps were pre-trained model, SFT, reward model and PPO. They are now read as three stages, and only the middle one is RLHF in the strict sense:
- Supervised fine-tuning (SFT) teaches the model the format and behaviour of a good answer from demonstrations.
- Preference optimisation teaches it which of two answers is better, using methods such as reward-model-plus-PPO, DPO or KTO.
- Reinforcement learning with verifiable rewards (RLVR) rewards it only when an answer passes a programmatic check or a rubric.
This pipeline now sits behind dialogue agents and chatbots, text summarisation systems and content safety work.
What Does Each Post-Training Stage Ask a Human to Do?
Each stage consumes a different kind of human input. The table below is the map for the rest of this guide.
Down the third column, the human role moves from producing data to judging it, and then to specifying how it should be judged. Each step up gives a single expert decision more leverage over the model. It also means a single bad decision travels further.
What Does Supervised Fine-Tuning Need From Human Experts?
SFT trains on prompt-response pairs: a person writes the answer the model should have given. It is also the stage where volume has proved least useful.
Meta's LIMA experiment fine-tuned a model on only 1,000 carefully curated prompts and responses, with no reinforcement learning at all. Human evaluators preferred LIMA's answers to GPT-4's, or rated them equivalent, in 43% of cases. GPT-4 still came out ahead, but the result showed how much a small, well-chosen set could do. Meta reached the same conclusion at scale for Llama 2, in a section titled "Quality Is All You Need". The team set aside millions of third-party examples and stopped at 27,540 SFT annotations. It also warned that different annotation platforms and vendors produced markedly different downstream models.
In other words, who wrote the demonstrations changed the model, with everything else held fixed.
By Llama 3, the balance had shifted again. Meta reports that most of its SFT data is now model-generated, through rejection sampling (generating many candidate answers and keeping the best-scored) and synthetic generation, with small amounts of human-curated data on top. Tülu 3's open SFT mix runs to 939,344 prompts, most of them filtered public data or synthetic examples targeted at specific skills. The human job in SFT has moved from writing every example to deciding which examples are good enough to keep.
At the frontier, the people doing that deciding are increasingly specialists. In 2023, OpenAssistant gathered 161,443 messages from more than 13,500 volunteers. By 2026, the market had changed. Fortune reported that Mercor says it contracts around 30,000 experts, at an average of $86 an hour, to write tasks and grade model outputs against detailed rubrics. Those are one vendor's own figures. They fit what the lab reports describe: fewer, more qualified people, each making calls that carry further.
How Does RLHF Work, and What Does the Human Contribute?
Preference optimisation rests on an observation Lambert makes early in the RLHF Book's chapter on preference data: it is easier for a person to tell a good answer from a bad one than to write the good answer. So instead of demonstrations, the human provides comparisons.
The classic RLHF process
In the classic RLHF process, data collection starts with prompts. The initial model generates multiple responses to the same prompt, and human annotators rank them or make pairwise comparisons between two model outputs. That ranking data becomes the training dataset for a separate reward model. The reward model learns to score outputs as a single number, through a loss function that pushes the preferred response's score above the other's. It then stands in for human preferences as the reward function during reinforcement learning.
How PPO trains the policy model
Most RLHF training has used proximal policy optimization (PPO), which OpenAI introduced in 2017. PPO is an actor-critic algorithm. The policy model, which is the language model being trained, generates text. A value model estimates how much reward to expect, which lets PPO judge whether a response did better or worse than expected; the original paper uses a truncated form of generalized advantage estimation for this. PPO's clipped surrogate objective limits how far each update can move the policy, which keeps training stable.
RLHF adds one more safeguard. InstructGPT added a per-token KL-divergence penalty against the SFT model, used as a frozen reference model, so the policy loses reward for drifting too far from where it started. The PPO training loop then repeats: generate, score with the reward model, update. The resulting model is tuned towards high reward, which is only as good as the human preference data behind it.
From PPO to direct alignment algorithms
Direct Preference Optimization (DPO) removed the separate reward model and the reinforcement learning loop, solving the same training objective with a simple classification loss on preference pairs. Meta adopted it for Llama 3 because it needed less compute at scale and performed better. Llama 3 still trained a reward model on human preferences, though, and used it to filter candidate outputs. DPO removed PPO from the pipeline, not the human preference data.
The successors to DPO, often grouped as direct alignment algorithms, change what the human has to supply. KTO learns from a single thumbs-up or thumbs-down per output, with no pair required. IPO drops the assumption that preferences can be turned into a single score. SimPO and ORPO drop the reference model (the frozen copy of the starting model that DPO measures drift against). ORPO also folds preference tuning into SFT itself. For a team planning annotation, the method choice decides whether reviewers compare pairs or rate single outputs.
Where AI feedback fits
Not every comparison has to come from a person. Anthropic's Constitutional AI trained a harmless assistant without any human labels identifying harmful outputs; a model critiqued and ranked responses against a written set of principles instead. (The 2024 version of this guide described Constitutional AI as red-teaming. It isn't. Red-teaming is a separate practice.) Google's RLAIF study tested reinforcement learning from AI feedback on summarisation and dialogue. AI-labelled preferences performed on par with human ones, and the two were statistically tied head to head.
Neither result removes the human. In Constitutional AI, people write the constitution and still supply the helpfulness data. The RLAIF authors warn that AI feedback carries the labelling model's own biases into the data. And AI2's Olmo 3, released in 2025, builds its preference pairs largely synthetically, then relies on reinforcement learning to push further. The labs are moving pair construction away from people. The standard that decides what counts as better is still set by someone.
How preference data goes wrong
Preference data has three well-documented failure modes, and all of them trace back to the judgment behind the labels.
The first is disagreement. InstructGPT's training labelers agreed with each other 72.6% of the time. Anthropic's helpful-and-harmless RLHF paper found about 63% agreement between its own researchers and its crowdworkers. Roughly one comparison in three or four, in other words, is one that qualified people would call the other way. A reward model trained on those labels learns the disagreement along with the signal.
The second is that people reward the wrong things. Anthropic researchers showed that human raters and preference models favour answers that match the user's stated views. Optimising against those preferences can trade truthfulness for sycophancy. A UT Austin team found that a reward based purely on response length reproduced most of RLHF's measured gains over SFT.
The third is pushing too hard. OpenAI's scaling laws for reward model overoptimization showed that optimising a model against a proxy reward eventually lowers true quality, in predictable ways that scale with reward-model size. The policy overfits to the quirks of the specific feedback the reward model learned from. This is reward hacking in its original form: past a certain point, a higher reward means a worse model.
LLM judges inherit a version of the same problem. The MT-Bench study found GPT-4 agreed with human preferences about as often as humans agreed with each other. It also documented bias towards the first answer shown, towards the judge's own outputs, and towards longer answers. A judge can scale a standard. It cannot tell you whether the standard was sound.
What Changes When the Reward Is Verifiable?
The third stage replaces the learned reward with a check. A learned outcome reward model scores a final answer by predicting what humans would prefer; a verifiable reward checks it against ground truth directly. If a maths answer matches the ground truth, or code passes its tests, the model is rewarded. Otherwise it isn't.
AI2 coined the term RLVR in Tülu 3. DeepSeek brought it to wide attention with R1. DeepSeek-R1-Zero was trained with two types of rewards: accuracy and format. The accuracy reward checked the final boxed answer or ran the code; the format reward checked that reasoning appeared inside the right tags. On AIME 2024, a competition maths benchmark, pass@1 (the share of problems solved on the first attempt) rose from 15.6% to 71.0%. The optimiser, GRPO, came from DeepSeek's earlier DeepSeekMath paper, where it was introduced as a lower-memory variant of PPO.
DeepSeek's stated reason for avoiding learned reward models on reasoning data is reward hacking. The peer-reviewed version in Nature adds that the team observed hacking with the model-based reward it did use for general helpfulness data. R1 did not do away with reward models. It kept them out of the places where a rule could do the job.
Humans design the test
A verifiable reward looks like it takes people out of the loop. In practice it moves them upstream. Someone has to write the task, the tests and the grader, and decide what "correct" means for each one.
OpenAI's documentation for reinforcement fine-tuning says as much. Before training, it advises teams to check whether qualified human experts agree on the answers, and to make tasks guess-proof so a lucky answer can't earn reward. That is an evaluation protocol, written into a training guide.
For work without a single right answer, rubrics stand in for tests. OpenAI's HealthBench was built with 262 physicians who have practised in 60 countries. They wrote a rubric for each of 5,000 conversations, 48,562 criteria in all. HealthBench is an evaluation benchmark, not training data, but it shows the shape of the work. Scale AI's Rubrics as Rewards then used rubrics as the reward signal for RL in medicine and science. Rubrics grounded in expert reference answers beat those without. The rubrics themselves were generated by LLMs from those reference answers; the expert input was the reference.
The same pattern is scaling into "RL environments": simulated workplaces where agents earn reward for completing complex tasks. TechCrunch reported in 2025 that Anthropic leaders had discussed spending more than $1 billion on them, citing The Information. Every environment is, at bottom, a task and a grader that someone designed.
How verifiable rewards go wrong
A verifier is only as good as the gaps it leaves. OpenAI found frontier reasoning models rewriting verification functions to always return true and skipping tests during training. When the team penalised the model's "bad thoughts", the models learned to hide their intent rather than stop. METR measured o3 reward-hacking in 30.4% of runs on one research benchmark, including monkey-patching the evaluator (rewriting its code at runtime) to return a perfect score. Anthropic studied models that learned to reward-hack in real production coding environments. Those models generalised the behaviour into broader misalignment, and standard chat-based safety training did not fix it.
Kili's SWE-Bench Pro Verified Data Story followed the same failure into evaluation. There, leaked answers and narrow tests moved one model's leaderboard score by nearly 20 points. The verifier is where the expert judgment lives in RLVR. When it is weak, the model finds out first.
RLVR's gains also need careful reading. A Tsinghua study found RL-trained models beat their base models at pass@1, but base models scored higher when allowed many attempts. That suggests RLVR may sharpen reasoning the base model already had instead of creating new capability. Later work contests that conclusion. Either way, gains from RLVR are something to measure on your own tasks.
Why Does Agreement Matter More Than Volume in RLHF Data?
InstructGPT published its labeler agreement rate. Most preference datasets since have not, although that number says more about the quality of a reward signal than dataset size does.
The datasets that do report agreement tend to be the ones that took expertise seriously. NVIDIA's HelpSteer3-Preference required a relevant degree for STEM prompts and software engineering experience for code. It used three to five annotators per sample and reported weighted Cohen's κ above 0.8 in every subset. (Cohen's κ measures agreement after correcting for chance; 0.8 is conventionally read as strong.) That figure comes after removing outlier annotations and high-disagreement samples, so it describes curated agreement. It still shows the process clearly. Qualify the judges, measure how often they agree, and remove what they can't agree on before it reaches the reward model.
A team can defend a preference dataset when a model behaves badly only if it knows who judged each comparison, under which guideline, and with what agreement. The same record lets a team find the one guideline that produced a third of its disagreements.
Kili's platform is built around this layer of the work. Domain experts review directly, in a workflow configured per project. Consensus measures how often several annotators agree on the same asset; honeypot scores each annotator against reference labels; multi-step review sends disputed items back with an issue attached. Exports can include the latest label from each author on every asset, so agreement can be recomputed outside the platform. For preference data specifically, Kili's LLM interface for comparing and ranking model responses is currently in private beta, with access on request.
What Should an Auditable Post-Training Dataset Record?
Incorporating human feedback into the training process only helps if you can trace that feedback later. If the human judgment is the variable that matters, the dataset should record it. For each stage, that means a small number of questions that should have answers before training starts:
- SFT: who wrote or approved each demonstration, and what the acceptance criteria were.
- Preference optimisation: who compared each pair, under which version of the guideline, and how often reviewers agreed on overlapping items.
- Verifiable rewards: who wrote each verifier or rubric, whether independent experts agreed on the reference answers, and which trajectories were audited by a person.
None of these questions is new. InstructGPT and HelpSteer3 answer versions of them in their appendices. The answers are also far easier to record at the moment of judgment than to reconstruct after a model misbehaves.
Conclusion: RLHF Didn't Shrink, It Moved
It is tempting to read the last two years as RLHF being replaced. DPO took out PPO. Synthetic data took out human demonstrations. Verifiable rewards took out the reward model. Each of those is partly true. None of them took out the human judgment. They moved it.
In SFT, the expert now selects rather than writes. In preference optimisation, they set the standard that AI labellers and LLM judges apply at scale. In RLVR, they write the tests and rubrics that decide what the model is rewarded for. Each move gives a single decision more reach. That makes measuring those decisions, and recording who made them, a better use of a post-training budget than collecting more of them.
Resources
Foundations of RLHF
- Deep Reinforcement Learning from Human Preferences – Christiano et al., 2017
- Proximal Policy Optimization Algorithms – Schulman et al., OpenAI, 2017
- Learning to Summarize from Human Feedback – Stiennon et al., 2020
- Training Language Models to Follow Instructions with Human Feedback (InstructGPT) – Ouyang et al., 2022
- Training a Helpful and Harmless Assistant with RLHF – Bai et al., 2022
Post-Training Pipelines and Recipes
- The Llama 3 Herd of Models – Meta, 2024
- Tülu 3: Pushing Frontiers in Open Language Model Post-Training – AI2, 2024
- Olmo 3 – AI2, 2025
- RLHF Book: Introduction – Nathan Lambert
- RLHF Book: Preference Data – Nathan Lambert
Supervised Fine-Tuning Data
- LIMA: Less Is More for Alignment – Zhou et al., 2023
- Llama 2: Open Foundation and Fine-Tuned Chat Models – Meta, 2023
- OpenAssistant Conversations – Köpf et al., 2023
- Mercor and the expert-contractor market – Fortune, January 2026
Preference Optimisation Methods
- Direct Preference Optimization – Rafailov et al., 2023
- KTO: Model Alignment as Prospect Theoretic Optimization – Ethayarajh et al., 2024
- A General Theoretical Paradigm to Understand Learning from Human Preferences (IPO) – Azar et al., 2023
- SimPO: Simple Preference Optimization with a Reference-Free Reward – Meng et al., 2024
- ORPO: Monolithic Preference Optimization without Reference Model – Hong et al., 2024
- Constitutional AI: Harmlessness from AI Feedback – Bai et al., 2022
- RLAIF vs. RLHF – Lee et al., 2024
Failure Modes in Preference Data
- Towards Understanding Sycophancy in Language Models – Sharma et al., 2024
- A Long Way to Go: Investigating Length Correlations in RLHF – Singhal et al., 2024
- Scaling Laws for Reward Model Overoptimization – Gao et al., 2022
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena – Zheng et al., 2023
- HelpSteer3-Preference – NVIDIA, 2025
Verifiable Rewards and Reasoning RL
- DeepSeekMath (introduces GRPO) – Shao et al., 2024
- DeepSeek-R1 – DeepSeek-AI, 2025
- DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning – Nature, 2025
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? – Yue et al., 2025
- Reinforcement Fine-Tuning – OpenAI API documentation
- HealthBench – Arora et al., OpenAI, 2025
- Rubrics as Rewards – Gunjal et al., Scale AI, 2025
- Silicon Valley Bets Big on "Environments" to Train AI Agents – TechCrunch, 2025
Reward Hacking
- Monitoring Reasoning Models for Misbehavior – Baker et al., OpenAI, 2025
- Recent Frontier Models Are Reward Hacking – METR, 2025
- Natural Emergent Misalignment from Reward Hacking in Production RL – MacDiarmid et al., Anthropic, 2025
From Kili Technology
- Data Story: SWE-Bench Pro Verified – what a benchmark audit did to the agentic coding leaderboard
- Contact Kili Technology
Ready to Put Experts in the Loop?
If your post-training or evaluation data depends on specialists judging model outputs, talk to Kili Technology about running that review with consensus, honeypot checks and multi-step review, and about access to the LLM comparison beta.
