.png)
_logo%201.svg)
AI Summary
- Published SFT pipelines use thousands of examples; filtered sets beat larger unfiltered ones.
- Set answer length in the guideline, because LLM judges reward longer answers.
- Ask experts to stay within facts the model can already support.
- Write the acceptance rule before labeling, and never let writers approve their own answers.
Introduction
When Meta built the supervised fine-tuning stage for Llama 2, it stopped at 27,540 annotations. More data was not the problem. Meta found that "different annotation platforms and vendors can result in markedly different downstream model performance." The same model, trained on demonstrations from different writers, came out different.
Supervised fine-tuning (SFT), also called instruction tuning, trains a pre-trained language model on prompts paired with target answers, so it learns to produce answers like the ones it was shown. An SFT dataset is that set of pairs. Training updates the model parameters. Full fine-tuning changes all of them; parameter-efficient fine-tuning methods such as LoRA (low-rank adaptation) train a small added set instead. Either way, a base model initially trained to predict text learns instruction following in the format the examples show. Published pipelines use thousands or tens of thousands of examples, so each answer carries weight. The people who write them decide what the model learns: its format, its tone, how long it talks and what it claims to know.
Our guide to SFT vs RLHF (reinforcement learning from human feedback) covers whether your SFT data should come from domain experts, and which stage of post-training needs them most. This guide starts after that decision. You have domain specific tasks, a base model (often one of the open foundation models) and a group of clinicians, underwriters, engineers or lawyers who will write the answers. What follows is how to choose those experts and what to tell them. Then come length limits, the rule for which answers enter the training set, and how to format the result.
What Makes a Good SFT Dataset?
Five properties come up across the published work as drivers of SFT success: enough examples but not too many, high quality per example, diverse prompts, a consistent style, and answers that fit what the base model already knows.
Size: thousands, not millions
Published SFT LLM pipelines use fewer examples than most teams expect. LIMA fine-tuned a 65B model on 1,000 curated prompt-response pairs, 200 of them written by the authors. Its ablation (an experiment that changes one factor at a time) found that "doubling the training set does not improve response quality" when the extra examples came from the same source. The authors concluded that prompt diversity mattered more than quantity. Llama 2 found that SFT annotations "in the order of tens of thousands" were enough. InstructGPT, trained to follow human instructions using human feedback, used about 13,000 training prompts for its SFT stage.
The same pattern holds for reasoning data. s1 fine-tuned Qwen2.5-32B-Instruct on 1,000 questions paired with reasoning traces, selected for difficulty, diversity and quality. Training on the full pool of 59,000 examples did "not offer substantial gains over our 1K selection." A Gemini model wrote the traces. The authors controlled which questions went in. If your SFT data includes reasoning content, spend expert time on choosing the questions, not only on writing the traces.
None of these results sets a universal dataset size. They show that which training examples you keep matters more than how many you have. For an expert team, start with a small pilot: it tells you whether the guideline works before you commit expert hours to thousands.
Quality: filtering beats adding
Data quality varies widely in large datasets. The AlpaGasus authors found the 52,000-example Alpaca set had "many low-quality instances with incorrect or irrelevant responses." They used an LLM rater to keep 9,229 examples that scored at least 4.5 out of 5. The model trained on that subset beat the one trained on all 52,000. Deita (ICLR 2024) selected about 6,000 examples on quality, complexity and diversity. It matched or beat baselines while using "over 10x less" data than they did.
Both papers filter after the fact, with model-based scorers. With expert writers, you can catch most of the same problems at the review step instead.
Diversity: cover the prompts users will send
LIMA's ablations found that more diverse prompts produced better answers than a homogeneous set of the same size. Deita found that random selection underperformed selection that accounted for diversity. For an expert dataset, diversity is a prompt-sampling decision. Draw prompts from the real distribution of requests (by topic, difficulty and user type) rather than letting each expert pick questions they find interesting. If the model will serve different domains, sample from each in proportion to expected use.
Then add edge cases on purpose: the rare but important requests, the ones with a trap, the ones the model should decline. If real requests are scarce, data augmentation can fill coverage gaps, through paraphrased prompts or prompt generation with an LLM seeded from real examples. The expert still writes or checks every answer.
Consistency: one voice, not twenty
LIMA's authors wrote their own answers in one assistant voice. They reported: "Preliminary experiments show that this consistent format generally improves model performance." The SCAR preprint found that human-written answers, authored by many different people, vary more in style than LLM-generated ones. Among training sets of similar quality, more consistent style produced better models. Consistent examples give the model one pattern to copy.
This follows from what SFT mostly teaches. LIMA's Superficial Alignment Hypothesis holds that a model's knowledge comes from pretraining, and that alignment teaches it "which subdistribution of formats should be used" with users. A token-level study, URIAL, found that "Most distribution shifts occur with stylistic tokens" when base and aligned models are compared. An SFT dataset is mostly a specification of style and format, so inconsistency between writers becomes inconsistency in the model.
Fit: answers the base model finds predictable
Harada et al. (EMNLP 2025) trained 1,070 SFT models. They combined a dozen base models of about 7B parameters with 10 different datasets, covering general chat, code generation, math and classic natural language processing tasks. They used both full and LoRA fine-tuning. Perplexity, a measure of how predictable the training text is to the base model before fine-tuning, predicted which data helped. Training data with lower perplexity for the base model "consistently leads to greater improvements," and "direct semantic similarity is not as strong a predictor as perplexity." Token length showed only a modest correlation. Weight changes in the middle layers tracked performance gains more closely than changes in the top or bottom layers.
The study supports the small-data results too: "a compact 1k instruction set already supplies sufficient signal for effective instruction-tuning." It also found that gains from the same instruction data vary substantially between base models. Different models need their own pilot.
These are correlations across public datasets, measured with benchmarks and LLM-judged evaluation results, not a test of expert-written data. They are consistent with the rest of the evidence. Answers that are low-perplexity for the base model use formats and facts it already knows, which is where LIMA's hypothesis and the new-knowledge result below both point. One practical use, which the paper doesn't test: score a pilot batch with the base model and send the highest-perplexity answers to a reviewer first. High perplexity can mean an unusual format, an unsupported fact or a hard prompt; the reviewer decides which.
Who Should Write SFT Answers, and How Do You Qualify Them?
Pick writers who can produce the answer to your quality bar in writing, then test that before they write training data. Domain knowledge and writing a clean, complete answer on a deadline are separate skills.
Published teams screened before they trained:
- InstructGPT hired about 40 contractors and gave them a screening test. "We selected labelers who performed well on this test," the authors wrote. The test included agreement with researchers "on a labeling task with detailed instructions."
- FinQA (EMNLP 2021) hired 11 finance professionals, including CPAs and MBAs, through Upwork. Candidates were interviewed by writing sample question-answer pairs, then trained.
- ExpertQA (NAACL 2024) required formal education in the field and "at least 3 years of work experience in their field." It recruited 484 participants across 32 fields.
- For Med-PaLM, a panel of five clinicians wrote exemplar answers.
The FinQA comparison shows why credentials matter for specialised work. On a random sample of 200 questions, two additional finance professionals averaged 91.16% execution accuracy (the share of answers whose calculation gave the correct result). General crowd workers on Mechanical Turk reached 50.68%.
A practical qualification step: give every candidate the same small set of prompts from your pilot. Have a senior expert review the answers against the rubric you will use in production (see below). Admit writers whose answers pass, and keep their qualification answers as reference examples for later checks.
Budget for time as well as rates. ExpertQA paid $15 an hour plus bonuses. Its experts spent 13.83 minutes on average per question-answer pair, and that was for checking and editing model answers, not writing from scratch. FinQA's finance experts were paid about $35 an hour on average.
What Should Writing Guidelines for Supervised Fine-Tuning Include?
The guideline is the specification of the target behavior the model learns. Write it before anyone writes an answer, test it in the pilot, and version it when it changes. Our general piece on data labeling guidelines covers structure; free-text answers need six things on top.
1. The target answer, described and shown
State the format (paragraphs, numbered steps, a fixed template), the register and the audience. OpenAssistant (NeurIPS 2023) asked contributors to "make assistant responses be polite, helpful, concise, friendly, and safety-aware." Your task is narrower, so your guideline should be more specific. Describe what a good discharge summary, claims note or code review comment looks like in your organisation.
Tone and safety are part of the target too. Llama 2's safety work began by collecting "adversarial prompts and safe demonstrations" and adding them to the general SFT data. If the model should decline some requests, the guideline needs examples of how.
Include worked examples, and vary them. Parmar et al. (EACL 2023) found that writers copy the examples in their instructions. Across 14 benchmarks, 72.7% of instruction examples shared one dominant pattern, and that pattern appeared in 60.5% of the collected data. Their advice: "Crowdsourcing instructions should be diverse."
2. Length limits
Give a maximum, and usually a minimum, for each task type. The next section explains why.
3. What the answer may claim
Tell experts to write answers the model can support from what it already knows, or from context included in the prompt. Gekhman et al. (EMNLP 2024) fine-tuned large language models on examples carrying knowledge the model lacks. The models learned those examples slowly, and once learned, "they linearly increase the model's tendency to hallucinate." An expert's instinct is to add the extra detail they know. In SFT data, that detail can teach the model to state facts it can't back up. If the model needs new knowledge, retrieval and other methods that supply facts at inference time are the safer route.
4. A rule on AI tools
Decide whether writers may use an LLM, and say so. Veselovsky et al. estimated that "33-46% of crowd workers used LLMs when completing the task" in a medical-summary study on Mechanical Turk (a preprint with a small sample). A follow-up study found LLM use was "reduced by about half by asking workers to not use LLMs" and by making LLMs harder to use, for example by disabling copy-paste. Workers told not to use LLMs did write lower-quality summaries. OpenAssistant's guidelines told contributors not to paste text from other sources without editing, ChatGPT included. Databricks' Dolly dataset card says its contributors, thousands of Databricks employees, were "explicitly instructed to avoid using generative AI." If you want experts to start from a model draft, make that an explicit editing task and supply the draft. Then you know what you're getting.
5. A way to set a prompt aside
Some prompts make poor training examples: the question is ambiguous, the evidence is contested, or it needs information the prompt doesn't give. Med-PaLM's clinicians dropped weak pairs rather than forcing them: "Clinicians then filtered out questions/answer pairs that they decided were not good examples," the paper says. Give writers a flag for prompts like these, and route flagged prompts to a senior expert.
6. A change log
Guideline changes made mid-project split your data into versions. Record the guideline version on each answer so you can find and re-review answers written under an older rule.
How Long Should Supervised Fine-Tuning Answers Be?
As long as your product needs, set as a number in the guideline, with a counter in front of the writer. Left alone, length drifts. Longer answers also tend to score better with automated judges whether or not their output quality is better.
The direct evidence on SFT length points one way, with a caveat. Long Is More for Alignment (ICML 2024) fine-tuned on the 1,000 Alpaca examples with the longest responses and beat LIMA and AlpaGasus under GPT-4 and PaLM-2 judges. The trained model's own answers got longer too. Deita found response length correlated with performance, "yet its effect is not substantial for datasets with already high quality." Long Is More's authors add their own warning: "It is important to note that length alone is not sufficient."
The caveat is the judges. Length-Controlled AlpacaEval is a preprint by members of the AlpacaEval team. In it, changing only the verbosity instruction in the prompt moved GPT-4 Turbo's win rate (how often the judge preferred its answers to a reference model's) "from 22.9% to 64.3%." Meta FAIR researchers working on length-following note that training algorithms tend to exploit this bias, and that "the expected length of responses is ill-defined in many queries." If no one specifies length, writers decide it one answer at a time. The evaluation that follows may then reward whoever wrote the most.
LIMA set bounds on its own data. Stack Exchange answers were dropped if they were "too short (less than 1200 characters), too long (more than 4096 characters)." Those bounds suited long-form community answers. Yours should come from where people will read the output:
- A field in a form or a structured extraction: a hard maximum in characters, often with an input type (number, date).
- A chat reply or customer-facing answer: a maximum that matches what the interface shows without scrolling, and a minimum that rules out one-line non-answers.
- A report section or long-form explanation: a range, plus a rule that reviewers reject padding even when an answer is within bounds.
Write the limit in characters if writers will see a counter, and keep it the same for every writer on the same task type. The trainer has a limit of its own, in tokens, covered in the formatting section below.
How Should Expert Answers Be Reviewed?
Every answer gets human review by a second qualified expert before it enters the training set, and no one reviews their own work. That is the core control. Agreement metrics help, but for free text they can't replace a reviewer.
The reviewer has to be an expert for the same reason the writer does. The False Promise of Imitating Proprietary LLMs (ICLR 2024) found that models trained on ChatGPT outputs were "adept at mimicking ChatGPT's style but not its factuality." The style fooled crowd raters without domain expertise. A fluent, well-formatted answer with one wrong dosage or one misread clause looks fine to a generalist.
A review workflow for SFT data has three parts.
A rubric the reviewer scores against
Split it into must-pass checks and a quality score. Must-pass checks are binary: factually correct for the domain, follows the required format, within the length limit, no claims beyond what the prompt and the model can support, no prohibited content. The quality score covers what is a matter of degree: completeness, clarity, tone. OpenAssistant used five-point Likert scales for quality labels. A reviewer who fails a must-pass check sends the answer back with the reason, and the original writer fixes it.
More than one review where the stakes are high
NVIDIA's HelpSteer2 (NeurIPS 2024 Datasets and Benchmarks) is a dataset for training reward models (models that score responses during RLHF). It gave every annotation "a minimum of two human reviews in addition to automated checks." For SFT, a second review step makes sense in two cases: high-risk categories (clinical, legal, financial advice), and the first weeks of a project, while you are still testing the guideline.
Overlap on a sample, used to find unclear guidelines
Send a share of prompts to two or more writers without letting them see each other's answers. For free text, the answers will differ even when both are correct, so text overlap is a weak acceptance signal for long answers. What overlap does show well is where the guideline is unclear. If two qualified experts write structurally different answers to the same prompt, the guideline allowed it. Fix the guideline, then re-review the answers written under the old version.
What Acceptance Threshold Should an SFT Answer Meet?
Decide the threshold before labeling starts, write it into the guideline, and apply it the same way to every writer. Published projects use several kinds:
HelpSteer2's rule removed about 10% of responses. Its quality checks raised helpfulness agreement (quadratic-weighted kappa, a chance-corrected agreement score) from 0.465 to 0.791. A rule that removes some data is doing its job.
For expert-written SFT answers, a workable acceptance rule has three layers:
- Accept an answer only if every must-pass check passes and the reviewer's quality score meets the bar you set in the guideline. Anything else goes back to the writer once; if it fails again, drop it.
- Qualify each writer on the pilot set, then check their work over time against reference answers that a senior expert has written or approved. A writer whose answers keep failing review needs feedback or removal, and their accepted answers need a second look.
- The dataset gets a final check: before each training run, a senior expert reads a random sample of accepted answers. Llama 2's team did a version of this, examining "a set of 180 examples" by hand and comparing human annotations with model samples. If the sample turns up problems the review step missed, tighten the rubric before adding more data.
For short, constrained answers (a number, a date, a label, a one-line extraction) overlap scores are meaningful. You can set an agreement threshold the same way you would for any labeling task. For multi-sentence answers, rely on the rubric.
How Should You Format the Finished SFT Dataset?
Data preparation ends with a file your trainer can read. Prepare data for the trainer only from accepted answers. Converting to the trainer's format is a small but critical step. Mistakes there are silent: the run completes, and the model learns the wrong thing.
Conversational format, system prompt and special tokens
Store one example per line; JSON Lines (JSONL) is the usual choice. Hugging Face's TRL SFTTrainer, part of its open-source post-training library, accepts two formats. In the conversational format, each example is a list of messages, and each message has a role and content. The simpler prompt-completion format holds one prompt and one answer. Keep the original datasets separate from the converted file, with the guideline version on each answer. Then you can re-export after a correction.
If the deployed model will run with a system prompt (standing instructions placed before every conversation), include the same one in the training examples. The model inputs at training then match the ones at inference. A chat template turns each conversation into one input sequence, adding special tokens that mark where each turn starts and ends. LIMA added "a special end-of-turn token (EOT) at the end of each utterance." Hugging Face warns that with the wrong control tokens, models would have "drastically worse performance." It also warns that adding special tokens again is "often incorrect or duplicated, hurting model performance." Use the base model's own template, and apply it once.
What the loss sees, and where examples get cut
The common practice is to compute the loss (the error signal the model learns from) on the answer only. Llama 2's team chose to "zero-out the loss on tokens from the user prompt." TRL does the same by default for prompt-completion data, ignoring the prompt tokens. That fits expert data, where the answer is the expert's work and the prompt is input data. It isn't a fixed rule. Shi et al. (NeurIPS 2024) found that also training on the instruction helps when instructions are long and outputs short, or when there are few training examples.
Trainers also cut examples at a maximum sequence length. In TRL, "Sequences are truncated to max_length during preparation." Llama 2 used "a sequence length of 4096 tokens" for SFT. LIMA trimmed texts longer than 2,048 tokens. A truncated example teaches the model an answer that stops mid-sentence. Check the token length of the longest prompt plus answer against the limit before training, and keep the character limit in your guideline well inside it. Packing, which places several short examples in one input sequence, saves compute but doesn't change this check.
Tool calling and structured outputs
If the model will call tools, the examples need the tool definitions as well as the conversation. In TRL, they go in a tools column next to the messages, "typically provided as JSON schemas." The assistant's turn holds the tool call. The same applies to structured outputs: if the model must return JSON, every accepted answer must be valid JSON in the target schema. For tool calling, the expert's job is choosing the right tool and arguments. That makes the must-pass check exact: the call matches the schema and the intent, or it doesn't.
Duplicates and a holdout set
Before training, remove exact and near duplicate examples, which give one answer extra weight. Then hold out a slice for evaluation, sampled the same way as the training data so it covers the same mix of prompt types. Evaluation results on that held-out set, scored against the same rubric your reviewers use, tell you whether final performance moved.
How Do You Run This Workflow in Kili?
Kili isn't used for training models; the SFT run happens in your own machine learning stack. Kili is where domain experts write and review the data before it gets there.
You import each prompt, with any context the writer needs, as a text asset. The expert writes the target answer in a transcription job, Kili's free-text task. Transcription jobs can restrict input to types such as numbers or dates, for short structured fields. The 21 October 2026 release (v26.2.13) added a maximum character count for transcription jobs, set in the Labeling Jobs Editor. Writers see a live counter (for example, 280/300). So far it appears only in the changelog; the transcription job pages don't describe it yet.
Review runs as a multi-step workflow: a labeling step followed by one or more review steps. For each review step, you choose where an answer goes if it doesn't pass review, and it returns to the original writer for correction. Enforce Step Separation, on by default, means a member can't work on more than one step for the same asset, so no one reviews their own answer. Consensus, enabled on the labeling step, sends a set percentage of assets to several writers blind. For transcription, Kili scores their agreement with the Dice coefficient, which measures text overlap. Honeypot, enabled on one workflow step, scores each writer's submission against a reference answer you upload through the SDK. For transcription, the same text-overlap rules apply. Both scores are most informative on short answers. On long answers, treat a low score as a reason for a reviewer to look before anyone rejects the answer. Accepted answers are then exported and converted into your trainer's format.
Conclusion: The Guideline Is the Model Spec
SFT teaches a model mostly how to answer, not what it knows. That makes the expert guideline, more than the expert's knowledge, the document that defines the model's behaviour: its structure, its length, its tone, what it will claim and when it declines. The review step is the only place that enforces that specification, answer by answer.
That changes where to spend effort. Teams that hire strong experts and give them a loose guideline get a dataset with twenty voices and a model that averages them. Other teams write the guideline first, test it on a pilot, set a length limit and an acceptance rule, and keep writers from approving their own work. They get a dataset small enough for experts to produce and consistent enough to teach one behaviour. As AI models draft more of the answers themselves, the same structure carries over. The expert's job moves toward editing and reviewing, and the guideline and the acceptance rule become the main things the expert team owns.
Resources
How much SFT data, and what kind
- LIMA: Less Is More for Alignment – Zhou et al. (2023), NeurIPS
- Llama 2: Open Foundation and Fine-Tuned Chat Models – Touvron et al. (2023), arXiv, Meta lab report
- Training Language Models to Follow Instructions with Human Feedback (InstructGPT) – Ouyang et al. (2022), NeurIPS, OpenAI
- s1: Simple Test-Time Scaling – Muennighoff et al. (2025), arXiv preprint
- AlpaGasus: Training a Better Alpaca with Fewer Data – Chen et al. (2024), ICLR
- What Makes Good Data for Alignment? (Deita) – Liu et al. (2024), ICLR
- Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality – Harada et al. (2025), EMNLP
Style, consistency and new knowledge
- SCAR: Style Consistency-Aware Response Ranking – Li et al. (2024), arXiv preprint
- The Unlocking Spell on Base LLMs (URIAL) – Lin et al. (2024), ICLR
- Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? – Gekhman et al. (2024), EMNLP
- The False Promise of Imitating Proprietary LLMs – Gudibande et al. (2024), ICLR
Response length
- Long Is More for Alignment – Zhao et al. (2024), ICML
- Length-Controlled AlpacaEval – Dubois et al. (2024), arXiv preprint
- Following Length Constraints in Instructions – Yuan et al. (2024), arXiv preprint, Meta FAIR
Recruiting and instructing expert writers
- FinQA: A Dataset of Numerical Reasoning over Financial Data – Chen et al. (2021), EMNLP
- ExpertQA: Expert-Curated Questions and Attributed Answers – Malaviya et al. (2024), NAACL
- Large Language Models Encode Clinical Knowledge (Med-PaLM) – Singhal et al. (2023), Nature, Google
- OpenAssistant Conversations – Köpf et al. (2023), NeurIPS Datasets and Benchmarks
- Don't Blame the Annotator: Bias Already Starts in the Annotation Instructions – Parmar et al. (2023), EACL
- databricks-dolly-15k dataset card – Databricks (2023), vendor self-report
LLM use by human writers
- Artificial Artificial Artificial Intelligence – Veselovsky, Horta Ribeiro & West (2023), arXiv preprint
- Prevalence and Prevention of Large Language Model Use in Crowd Work – Veselovsky et al. (2023), arXiv preprint
Review and acceptance
- HelpSteer2: Open-Source Dataset for Training Top-Performing Reward Models – Wang et al. (2024), NeurIPS Datasets and Benchmarks, NVIDIA
Formatting the dataset
- SFT Trainer – Hugging Face TRL documentation (read 9 Nov 2026)
- Chat templates – Hugging Face Transformers documentation (read 9 Nov 2026)
- Instruction Tuning With Loss Over Instructions – Shi et al. (2024), NeurIPS
Kili
- SFT vs RLHF Explained – which post-training stage needs experts who write, and which needs experts who judge
- Data Labeling Guidelines: Dos, Don'ts and Pro Tips
- Kili changelog, 21 October 2026 (v26.2.13) – maximum character count and live counter for transcription jobs
- Talk to the Kili team
Ready to Put Your Experts on Your SFT Data?
Kili lets domain experts write and review fine-tuning data in multi-step workflows, with consensus and honeypot checks configured in each project's workflow. Talk to the Kili team about setting up an expert-in-the-loop SFT project.
