.png)
_logo%201.svg)
AI Summary
- SFT needs someone who can write the answer; RLHF needs someone who can rank answers.
- Llama 2's SFT outputs rivalled human-written demonstrations, so Meta shifted annotation to preferences.
- Crowd raters missed the factual errors in ChatGPT imitations.
- Qualify every judge on a gold set before training on their labels.
Introduction
In 2025, OpenAI asked physicians to write the best possible reply to a set of health conversations. In one condition, they had model answers to work from. Against models from September 2024, the physicians improved the answers. Against models from April 2025, the HealthBench announcement reports that "physicians' responses no longer improved over the responses from the newer models." It is one benchmark, run by one lab on its own models. But the benchmark still depends on physicians in another role. Its 262 physicians wrote the 48,562 unique rubric criteria used to grade every model answer.
Those two roles map onto the two halves of post-training. Supervised fine-tuning (SFT) teaches a model from demonstrations: someone writes the answer, and the model learns to imitate it. Reinforcement learning from human feedback (RLHF) teaches it from comparisons. RLHF incorporates human feedback as judgments: someone looks at two answers and says which is better. The comparison is usually presented as a trade-off between cost and capability. SFT is cheaper and simpler; RLHF is more expensive and handles open-ended quality better.
For AI teams deciding who to hire, a more useful framing is the staffing question. Demonstrations need someone who can do the task. Preferences need someone who can judge it. Those are different skills, and they are not always held by the same person. The physicians in HealthBench were no longer the best writers in the room. They were still the only qualified judges.
This guide explains the difference between SFT and RLHF, what each asks of an expert, and how to check your judges before their labels train anything.
SFT vs RLHF: What Is the Difference?
Supervised fine-tuning (SFT) trains a pre-trained model on labeled examples, prompts paired with target responses, so it learns to imitate the demonstrated answer. In machine learning terms, it is supervised learning on labeled data, and it is usually the initial training stage after pretraining. The base model already has a broad understanding of language and impressive capabilities. SFT teaches it format adherence and shapes the model's behavior on specific tasks.
Mechanically, the model reads each prompt and its target answer in a forward pass, and the loss function is the standard cross-entropy loss on the next token. Meta's Llama 2 report says it backpropagates "only on answer tokens," so the model learns the response, not the prompt. The data unit is a set of labeled input-output pairs: one prompt, one good answer. Teams typically create anywhere from 1,000 demonstrations (LIMA) to tens of thousands (Llama 2), then fine-tune the model on them.
RLHF trains the model on human judgments of its own outputs. The RLHF process most teams know comes from OpenAI's InstructGPT paper. Step one is SFT: "Collect demonstration data, and train a supervised policy." Step two is collecting human feedback. People rank several model outputs for the same prompt, and that ranking data drives reward model training. The trained reward model acts as a learned reward function that predicts which answer a person would prefer. Step three is reinforcement learning (training by trial and error against a reward). PPO (Proximal Policy Optimization) updates the policy model, meaning the model being trained, to maximise that reward signal. PPO is an actor-critic algorithm: alongside the policy, it trains a value model that estimates how good each step is. The objective function also penalises drift from the initial model, "to mitigate over-optimization of the reward model." InstructGPT used about 13,000 prompts for SFT, 33,000 for the reward model and 31,000 for PPO.
RLHF dates from 2017, but InstructGPT made this three-step recipe the standard for language models. OpenAI released ChatGPT in November 2022. It was trained "using the same methods as InstructGPT," with slight differences in its data collection setup.
Direct preference optimisation methods such as DPO drop the separate reward model and train on the human preference data directly. The learning process changes, but for staffing purposes they sit on the RLHF side: the human still compares answers instead of writing them.
SFT shows the model one good answer per prompt. It never sees a worse alternative, so it gets no signal about which mistakes matter more. Comparisons supply that signal, which is why RLHF is used to align outputs in open-ended text generation, where qualities like helpfulness and harmlessness are hard to demonstrate.
SFT is simpler to run: the training process uses one model and one loss. ReMax describes PPO-based RLHF as "leading to laborious hyper-parameter tuning and significant computation burdens," and the DPO paper calls RLHF "a complex and often unstable procedure."
The results favour using both. In InstructGPT, people preferred outputs from the 1.3B model to those of the 175B GPT-3. The paper credits the gains to a ladder: prompting, then SFT, then PPO on comparison data. Its models also "generate about 25% fewer toxic outputs than GPT-3 when prompted to be respectful." But the same paper found the model "does not significantly improve over GPT-3 on the Winogender" gender-bias benchmark. It also reported "performance regressions compared to GPT-3 on certain public NLP datasets." These included natural language understanding tasks such as SQuAD, a reading-comprehension benchmark. OpenAI called this cost an alignment tax. Kirk et al. found that "RLHF generalises better than SFT to new inputs," especially as inputs drift further from training. The cost was much less varied output. In a controlled ablation (an experiment that changes one factor at a time), Ivison et al. found preference learning can improve performance. "Our best models improve on the SFT model by over 8 points" on instruction following and truthfulness, they reported. Factuality stayed within about one point, so preference learning did not help it. Preference learning shapes how a model answers far more than what it knows. Published pipelines, from InstructGPT and Llama 2 to Llama 3, Tülu 3 and DeepSeek-R1, all run SFT before further training with preferences or reinforcement learning.
When should you use SFT, RLHF, or both?
Start with SFT when the task has a fixed format and a clear right answer, such as extracting fields from a document or following a report template. Curate the data for that task rather than reusing a general set. LIMA got strong results from 1,000 curated examples, and Llama 2 chose "fewer but higher-quality examples from our own vendor-based annotation efforts" over large third-party sets. Add RLHF or DPO when quality is a matter of judgment: helpfulness, tone, safety, what to leave out. Add it too when the model must handle inputs unlike its training data, where Kirk et al. found the RLHF advantage grows. If a program can check the answer, reinforcement learning with verifiable rewards is a third option.
Kili's 2026 RLHF guide maps the full post-training pipeline. This article stays on one question: who should produce each kind of data.
Why Is SFT vs RLHF a Staffing Question?
RLHF was invented for tasks where people can judge more easily than they can perform. In the 2017 paper behind modern RLHF, Christiano et al. describe settings where "we can only recognize the desired behavior, but not necessarily demonstrate it." Incorporating human feedback this way let their agents solve complex tasks with feedback on less than 1% of their interactions. Jan Leike, a co-author, later made it a principle in his alignment essay: "Evaluation is easier than generation." His example: a NeurIPS paper can take over 1,000 hours to write, while reviewing it takes a few.
The same gap shows up in models. OpenAI's self-critiquing study puts it plainly: "Recognizing errors is easier than avoiding them."
Meta acted on this when it built Llama 2. Its SFT model's outputs were "often competitive with SFT data handwritten by human annotators." So Meta stopped SFT annotation at 27,540 examples. It chose to "devote more annotation effort to preference-based annotation for RLHF," and eventually collected more than a million comparisons. The LIMA team hand-wrote 200 of its 1,000 training examples and reached the same limit from the other side. It wrote that "the mental effort in constructing such examples is significant and difficult to scale up."
The two stages need two different job descriptions:
- A demonstrator can produce a complete answer at the quality bar, consistently, in the format the model should learn.
- A judge can look at two answers, tell which is better, and stay consistent across hundreds of comparisons and with other judges.
A senior claims adjuster may judge two claim summaries well and write them slowly. A clinician can spot a dangerous omission in a patient-facing answer without being the best person to draft it. Treating "domain experts" as one pool for both stages hides this distinction.
Is the Domain Expert's Writing Job Shrinking?
For general-purpose models that generate text, yes.
On medical questions, physicians in Google's Med-PaLM 2 study "preferred Med-PaLM 2 answers to those produced by physicians" on eight of nine clinical-utility axes. In the HealthBench paper, physicians writing without a model reference "appeared relatively weak." The authors say to read that result with caution. With a model reference, their rewrites stopped improving on the newest models. Both are labs reporting on their own models, but they point the same way.
Frontier pipelines have changed to match. Llama 3 builds most of its SFT data from the model's own outputs, filtered by rejection sampling (generating several answers and keeping the best). It adds synthetic data and "small amounts of human-curated data." Human input goes into preferences, and annotators can improve the winning answer: "Annotators edit the chosen response directly or prompt the model with feedback." That gives a three-level ranking, edited over chosen over rejected. DeepSeek-R1 bootstraps its SFT stage from model outputs too, with human annotators rewriting the reasoning into a more natural style and verifying the results. Ai2's Tülu 3 mixes public human-written datasets with synthetic prompts. A large language model (LLM), not people, labels its preferences.
So the demonstrator role is turning into an editor role. The expert starts from a model draft, fixes it, and in doing so produces a better demonstration and a preference pair at once. The expert's job is moving from author to editor and judge. That shift is welcome when experts are scarce. It also moves the risk. If the person editing or ranking can't see what is wrong with a fluent answer, nothing downstream will.
If Judging Is Easier Than Writing, Why Can't Anyone Do It?
Because "easier" describes the effort per item. It says nothing about who is qualified. A comparison takes less effort than a demonstration only for someone who knows what a good answer looks like. Without that knowledge, a judge falls back on surface signals such as length and confidence, and the model learns those signals. A lay rater can rank a fluent wrong answer above a technically correct one.
The clearest case is The False Promise of Imitating Proprietary LLMs. UC Berkeley researchers fine-tuned open models on ChatGPT outputs. Crowd raters scored the models trained this way as competitive with ChatGPT. Targeted benchmarks showed they had closed almost none of the capability gap. They had copied ChatGPT's style and little of its factual accuracy. The authors' conclusion: "crowd workers without domain expertise or significant time investments can easily be deceived."
Other studies show the same failure in finer detail:
- Human Feedback is not Gold Standard found "the assertiveness of an output skews the perceived rate of factuality errors." Confident answers look more correct to raters, and the authors report preliminary evidence that preference training makes models more assertive.
- Anthropic's sycophancy study showed that humans and preference models sometimes "prefer convincingly-written sycophantic responses over correct ones," meaning answers that tell users what they want to hear.
- In an ACL 2023 study of long-form answers, 13 domain experts across seven fields reached an estimated pairwise agreement of about 80%. The crowd raters behind OpenAI's WebGPT leaned on surface features: "WebGPT annotators are far more likely to mark conciseness and specificity as decisive factors."
- When Anthropic built its HH-RLHF dataset, it wrote: "we did not expect crowdworkers to fact-check our models significantly." Its crowdworkers agreed with Anthropic researchers only about 63% of the time.
The Open Problems and Fundamental Limitations of RLHF survey lists this as a fundamental limit: "Humans cannot evaluate performance on difficult tasks well."
RLHF performance depends on high-quality preference data, and that quality depends on who judges. Preference data is how RLHF-trained models learn human values such as helpfulness, so a model ends up reflecting its judges. InstructGPT's authors say their model is aligned to "the stated preferences of a specific group of people."
Can an LLM judge replace the expert?
On general chat, often yes. The MT-Bench study, built on multi-turn chat questions, found GPT-4 reached "over 80% agreement, the same level of agreement between humans" (counting only non-tied votes). Ivison et al. found synthetic preference data generally beat human-annotated datasets, and Zephyr was aligned with DPO on AI feedback, using GPT-4 scores instead of human annotation.
In expert domains, the picture changes. Szymanski et al. asked dietitians and clinical psychologists to compare model outputs. The experts agreed with each other 75% and 72% of the time. They agreed with an expert-persona LLM judge only 68% and 64% of the time. Lay users agreed with the LLM judges more often, 76% to 80% of the time, so the judges sided with the non-experts. Tseng et al. tested LLMs on finance, biomedical and legal annotation, a common natural language processing workload. GPT-4o averaged 67.8%, and the authors concluded that LLMs "do not present as a direct alternative for annotation tasks requiring domain expertise."
In HealthBench, a model grader agreed with physicians about as often as physicians agreed with each other. But it graded against rubric criteria the physicians had written. The expert's judgment was still there, written down in advance as criteria.
Medical evaluation designs already split who judges what. In Google's original Med-PaLM study, clinicians rated scientific consensus, potential harm and bias. Lay raters were asked only whether an answer was helpful and addressed the question's intent. Preference data in any expert domain should follow the same split. Kili's Who Judges the Judge? data story shows how often expert and crowd majorities pick opposite winners.
Judging is also not fast when it is done properly. OpenAI's GDPval used professionals with an average of 14 years' experience, and a single blinded comparison on its gold subset took over an hour. For CriticGPT, trainers with a median of five years' Python experience spent about 50 minutes per example. Per item, comparison takes less effort than writing. It still needs a qualified person.
When Does SFT Still Need an Expert Who Can Do the Task?
When the model cannot yet produce a draft worth editing. If an expert rewrites most of every model answer, they are a demonstrator in practice, and the data should get the same consistency checks as demonstrations. Cold starts are the most common case, and one caution applies.
Cold starts and new formats
DeepSeek-R1-Zero, trained with reinforcement learning and no SFT, produced reasoning that was hard to read and mixed languages. DeepSeek fixed that with a cold-start SFT stage (a small round of SFT before reinforcement learning begins) on "thousands" of curated examples. Chu et al. reached a similar conclusion in a controlled study: "SFT remains essential for effective RL training," because it stabilises the output format that RL then builds on. If your task has a structure the model has never produced (a specific report template, a coding convention, a regulated disclosure), someone who can write that structure has to demonstrate it first.
A warning about demonstrations that teach new facts
Experts writing a training dataset of demonstrations naturally include what they know. That can backfire. Gekhman et al. studied fine-tuning examples that contain knowledge the model doesn't already have. The model learns them slowly, and once learned, "they linearly increase the model's tendency to hallucinate." Our reading: use demonstrations to teach behaviour and format the model can ground in what it already knows. If the model lacks the facts, retrieval (looking facts up in a document store at answer time) or further pretraining is a safer route than SFT. During post-training, the expert's job is to check that answers stay within what the model can support.
How Should You Staff SFT and RLHF in Practice?
Staff by role. Most post-training projects need some mix of four roles:
Four questions decide the mix:
- Can the current model draft an acceptable answer? If not, you need demonstrators. If mostly, you need editors. If yes, you need judges.
- Can a crowd or an LLM judge agree with your experts on this task? Test it on a small expert-labelled set. On general chat the answer may be yes. In clinical, legal or financial work, the studies above suggest it is often no.
- Can the quality bar be written down as criteria? If it can, your most senior experts write rubrics, and judges or graders apply them. DeepMind's work on Sparrow dialogue agents broke good behaviour into written rules to "collect more targeted human judgements." Kili's RLVR guide covers reward hacking, where verifiers and rubrics get gamed.
- Where does substitution fail? Spend your scarcest experts on the contested, high-stakes comparisons that nobody else can call. CriticGPT showed that human and model teams beat humans alone, so give judges model assistance rather than more volume. Choosing which items reach experts helps too: an active preference learning method for DPO improved "both the rate of learning and final performance."
Expert hours are expensive. Bloomberg Government reported recruiters for AI training work "paying up to $400 an hour for civil rights attorneys," a ceiling rather than a typical rate. At rates like that, an hour spent writing an answer the model could have drafted buys less than an hour spent judging what only that expert can judge.
How Do You Know Your Expert Judges Are Sound?
You measure them. Domain expertise is necessary for judging, but experts don't agree perfectly either. Experts reached an estimated 80% agreement in the long-form answer study, 75% and 72% in Szymanski's dietetics and mental-health panels, and 71% in GDPval. InstructGPT's screened labelers agreed with each other 72.6% of the time. A preference label from one expert is a single vote, not ground truth.
A workable policy, consistent with the threshold Kili uses for LLM-judge validation:
- Build a gold set first. Have senior experts label 50 to 100 items with settled answers. The golden datasets guide covers how to choose and maintain them.
- Qualify each judge against it. Aim for at least 75% agreement with the gold labels. Read it alongside a chance-corrected figure such as Cohen's kappa (agreement adjusted for what chance alone would produce), since 75% on a skewed label set can be close to chance.
- Overlap a share of live items, blind. Send a sample to two or more judges without showing them each other's answers, and track agreement by task type.
- Drop or adjudicate splits. A comparison where qualified experts disagree is a weak training signal. Remove it or have a senior reviewer decide, and record who decided.
- Separate the reviewer from the labeller. No one should approve their own work.
- Watch the trend. Falling agreement on one slice usually means the guideline is unclear there, not that the experts got worse.
The same checks apply to demonstrators and editors.
Where Kili fits
Kili doesn't train the model. The SFT and RLHF training runs in your own stack. Kili is where domain experts write, correct and judge that data before it reaches training. For preference data, each prompt and its candidate responses are imported as a text asset. The expert records which response is better, and by how much, in a single-choice classification job (for example, from "A much better" to "B much better"). For SFT data, the prompt and the model's draft are imported as a text asset. The expert writes the target answer, or a corrected version of the draft, in a transcription job. Consensus, enabled on a labelling step, measures how often experts agree on the same asset without seeing each other's answers. Honeypot scores each expert's work against assets with known answers, which is the gold-set check in step 2. Enforce Step Separation, on by default, means no one reviews their own work. Kili's LLM labeling projects, designed for pairwise comparison of model responses, are in private beta; ask the Kili team for access.
Conclusion: Hire for the Judgment
The SFT vs RLHF debate assumes the hard part of building an artificial intelligence system is choosing a training method. For teams with scarce experts, the harder part is deciding what those experts spend their hours on.
As models get better at drafting, the demonstrator's job narrows to cold starts, new formats and edits, while the judge's job grows. Judging also becomes the place where errors enter the weights. A demonstrator's mistake is visible in the training file. A judge's mistake is a preference for the confident, concise, wrong answer, and the model learns to produce more of them.
So budget for judgment the way you would budget for any critical measurement. Know who your judges are, qualify them against a gold set, measure how often they agree, and keep a record of who decided what under which standard. Teams that hire experts as writers pay premium rates for the part a model increasingly does. Teams that hire them as measured judges spend expert time where nothing else can stand in.
Resources
SFT and RLHF foundations
- Training Language Models to Follow Instructions with Human Feedback (InstructGPT) – Ouyang et al. (2022), NeurIPS, OpenAI
- Introducing ChatGPT – OpenAI (2022), lab announcement
- Direct Preference Optimization: Your Language Model Is Secretly a Reward Model – Rafailov et al. (2023), NeurIPS
- ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models – Li et al. (2024), ICML
- Deep Reinforcement Learning from Human Preferences – Christiano et al. (2017), NeurIPS
- Llama 2: Open Foundation and Fine-Tuned Chat Models – Touvron et al. (2023), arXiv, Meta lab report
- LIMA: Less Is More for Alignment – Zhou et al. (2023), NeurIPS
- Understanding the Effects of RLHF on LLM Generalisation and Diversity – Kirk et al. (2024), ICLR
- Unpacking DPO and PPO – Ivison et al. (2024), NeurIPS
- SFT Memorizes, RL Generalizes – Chu et al. (2025), ICML (rule-based rewards)
How modern pipelines use human data
- The Llama 3 Herd of Models – Grattafiori, Dubey et al. (2024), arXiv, Meta lab report
- DeepSeek-R1 – DeepSeek-AI (2025), arXiv; published in Nature
- Tülu 3: Pushing Frontiers in Open Language Model Post-Training – Lambert et al. (2024), arXiv, Ai2
- Zephyr: Direct Distillation of LM Alignment – Tunstall et al. (2023), arXiv, Hugging Face
- Training a Helpful and Harmless Assistant with RLHF – Bai et al. (2022), arXiv, Anthropic
Judging vs generating
- Self-Critiquing Models for Assisting Human Evaluators – Saunders et al. (2022), arXiv, OpenAI
- Why I'm Optimistic About Our Alignment Approach – Jan Leike (2022), researcher blog
- Open Problems and Fundamental Limitations of RLHF – Casper et al. (2023), TMLR
- LLM Critics Help Catch LLM Bugs (CriticGPT) – McAleese et al. (2024), arXiv, OpenAI
- Improving Alignment of Dialogue Agents via Targeted Human Judgements (Sparrow) – Glaese et al. (2022), arXiv, DeepMind
- Active Preference Learning for Large Language Models – Muldrew et al. (2024), ICML
Where non-expert judgment fails
- The False Promise of Imitating Proprietary LLMs – Gudibande et al. (2024), ICLR
- Human Feedback is not Gold Standard – Hosking, Blunsom & Bartolo (2024), ICLR
- Towards Understanding Sycophancy in Language Models – Sharma et al. (2024), ICLR, Anthropic
- A Critical Evaluation of Evaluations for Long-form Question Answering – Xu et al. (2023), ACL
- Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? – Gekhman et al. (2024), EMNLP
LLM judges and expert evaluation
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena – Zheng et al. (2023), NeurIPS Datasets and Benchmarks
- Limitations of the LLM-as-a-Judge Approach in Expert Knowledge Tasks – Szymanski et al. (2024), arXiv
- Are Expert-Level Language Models Expert-Level Annotators? – Tseng et al. (2024), arXiv
- Introducing HealthBench – OpenAI (2025), lab announcement
- HealthBench: Evaluating Large Language Models Towards Improved Human Health – Arora et al. (2025), arXiv, OpenAI
- Large Language Models Encode Clinical Knowledge (Med-PaLM) – Singhal et al. (2023), Nature, Google
- Toward Expert-Level Medical Question Answering (Med-PaLM 2) – Singhal et al. (2025), Nature Medicine, Google
- GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks – Patwardhan et al. (2025), arXiv, OpenAI
The market for expert data (recruiter figures)
- Doctors and Lawyers Are Joining AI's Gig Economy – Bloomberg Government (2026)
Kili guides
- 2026 RLHF Guide – where SFT, preference optimisation and RL fit in post-training
- Who Judges the Judge? – expert vs crowd labels, and raw vs chance-corrected agreement
- How to Build Golden Datasets for Testing, Fine-Tuning and Evaluating AI Models
- RLVR Explained: Reinforcement Learning with Verifiable Rewards – how verifiers and rubrics get gamed
- Talk to the Kili team
Ready to Put Your Experts Where They Count?
Kili lets your domain experts write, correct and rank post-training data, with consensus and honeypot checks configured in each project's workflow. Talk to the Kili team about setting up an expert-in-the-loop SFT or preference data project.
