.png)
_logo%201.svg)
AI Summary
- Distillation trains a small model on a larger model's outputs.
- Students copied ChatGPT's style but not its factuality (Gudibande et al., ICLR 2024).
- Average accuracy can rise while confident rare-class errors transfer.
- Teacher traits survived "rigorous and validated filtering" in a 2026 Nature study.
- Audit teacher outputs before training.
Introduction
Model distillation is how most small, fast language models get their abilities. A large "teacher" model produces outputs, and a smaller "student" model is trained to reproduce them. DeepSeek, for example, fine-tuned six open models on roughly 800,000 samples curated with its R1 reasoning model. The models ranged from 1.5 billion to 70 billion parameters. OpenAI and AWS both sell distillation as a managed workflow.
The idea goes back to a 2015 paper by Geoffrey Hinton, Oriol Vinyals and Jeff Dean. Their case for it rests on the wrong answers. A trained classifier assigns small probabilities to every wrong answer, and those probabilities say something real. An image of a BMW might be mistaken for a garbage truck, but far less often for a carrot. In their words, "the relative probabilities of incorrect answers tell us a lot" about how the teacher generalises. Distillation passes that information on.
It passes on the teacher's misjudgements by the same route. A student learns the teacher's view of the world, including the parts where the teacher is confidently wrong. Much of what it inherits never shows up in an average benchmark score. And once an error is in the student's weights, no amount of student-side training tells you it came from the teacher.
This guide covers how distillation works, why teams use it and where it falls short. It then shows where domain experts should check the teacher's outputs before they become training data.
What Is Model Distillation?
Model distillation is a training method in which a smaller "student" model learns to reproduce the outputs of a larger "teacher" model. The student approaches the teacher's performance on a task at a fraction of the size and inference cost. The technique's formal name is knowledge distillation. In large language models (LLMs), it is often called LLM distillation or simply AI distillation.
The concept is knowledge transfer from a large model to a smaller model. The student model learns from the teacher's answers, not from the original training data alone. In the classic setup, it learns from the teacher's full probability distribution over possible answers. The teacher becomes the source of the labels, and the distilled model is a new model shaped by them.
In LLM practice, distillation usually means something concrete. You send a set of prompts to a strong model, collect its model outputs, and fine-tune a smaller model on the prompt and response pairs. Anthropic's definition is as short as any: "training a less capable model on the outputs of a stronger one."
How Does Model Distillation Work?
Every method involves training the student to mimic some part of the teacher. They differ in what the student sees.
Soft targets, temperature and the loss function
In Hinton, Vinyals and Dean's formulation, the student trains on the teacher's class probabilities, which they call soft targets. Ordinary training uses hard one-hot labels instead, where one class is marked correct and every other class gets zero. To make the soft targets informative, the teacher's softmax (the final layer that turns scores into probabilities) runs at a raised temperature T. A higher T flattens the distribution. The small probabilities on wrong classes become large enough for the student to learn from. The student trains at the same temperature, then runs at T = 1.
Gou et al.'s survey (IJCV 2021) sets out the usual loss function, which combines two terms. One term matches the teacher model's probability distribution. The other is a standard cross-entropy term (the usual classification loss) against the correct answer. A weighting parameter decides how much the student trusts the teacher's predictions over the ground-truth label. So the student learns from probability distributions as well as single labels.
Soft targets carry more information per example than hard labels, so the student can often train on much less data than the teacher did. The paper's experiment on MNIST, a dataset of handwritten digits, demonstrates this. The authors removed every example of the digit 3 from the student's training set. The student still learned a good deal about 3s from the teacher's soft targets on other digits. As the authors put it, "3 is a mythical digit that it has never seen."
The risk travels the same way. Whatever the teacher believes about 3s, right or wrong, is what the student learns.
What are the main types of model distillation?
Gou et al. group distillation by the kind of knowledge transferred. Response-based distillation trains the student on the teacher's final outputs, as above. Feature-based distillation also matches intermediate representations, such as the hidden states (internal activations) inside the network. Relation-based distillation transfers the relationships the teacher sees between its layers, or between data samples.
DistilBERT (Sanh et al., 2019) uses the first two. Its training mixes three losses: language modelling, distillation, and a cosine loss that aligns the student's and teacher's hidden states. The result was a model 40% smaller and 60% faster than BERT that kept 97% of its language understanding. It is small enough to run on edge devices, with far lower hardware requirements.
Sequence-level distillation: training on teacher outputs
Soft targets work well for classification. For generation tasks, Kim and Rush (EMNLP 2016) showed that distillation still works when the student trains on the sequences the teacher actually generates. Their best translation student ran about 10 times faster than its teacher. With pruning, it had 13 times fewer parameters, for a loss of 0.4 BLEU (a standard translation-quality score).
This is the ancestor of how most LLM distillation works now. The DeepSeek-R1 paper fine-tuned Qwen2.5 and Llama models on about 800,000 samples curated with R1 to pass on its reasoning capabilities. OpenAI and Amazon Bedrock sell the same loop as a managed service. You collect a frontier model's responses to your prompts, then fine-tune a smaller model on them.
In every case the training examples are the teacher's output, and nothing else.
Distilling step by step: training on rationales
Distilling step-by-step (Hsieh et al., Findings of ACL 2023) asks the teacher for more than the final answer. It also asks for a rationale: a natural-language explanation that supports the answer by connecting the input to the output. The authors extract rationales from a 540-billion-parameter PaLM model using few-shot chain-of-thought prompting. The prompt pairs a handful of input examples with their rationales and labels, and the model follows the pattern for new inputs. The student is then trained on two tasks: predict the label, and generate the rationale.
The results favour the small model. On the ANLI benchmark, a 770-million-parameter T5 student trained this way outperformed the few-shot prompted PaLM teacher using only 80% of the dataset. In other words, the student beat its own teacher's accuracy on that task. Standard fine-tuning of the same T5 model did not achieve that even with the full dataset.
The method also helps when labels are scarce. That matters because conventional distillation needs "large amounts of unlabeled data." The paper reports that small models still matched or beat the LLM "when only unlabeled data is present."
The rationale is still the teacher's, though. If its reasoning is wrong, the student learns wrong reasoning together with the right label.
On-policy distillation
Training on teacher outputs has a known weakness. At inference time, the student generates its own text, which can drift into states the teacher's examples never covered. On-policy distillation (Agarwal et al., ICLR 2024) targets this "distribution mismatch between output sequences seen during training" and at inference. The teacher scores the student's own outputs, token by token. But the teacher is still the judge, so its errors now reach the student more efficiently.
Why Do Teams Use LLM Distillation?
The reasons are cost, speed and control. A distilled model can be a tenth or a hundredth of the model size of its teacher. It can serve one narrow task at a much lower cost per request and lower latency. It can also run on your own infrastructure if data can't leave it. The incredible power of frontier models is of little use to a system that has to answer in milliseconds on a phone.
Real-time systems are a natural fit. Streaming speech recognition has to transcribe as the user speaks, without the future context a full-utterance model can use. Google researchers (Doutre et al.) distilled a non-streaming teacher into a streaming student, using large unlabelled audio datasets. In French, this cut the word error rate by 16.4% relative to a streaming baseline.
Distillation is also cheap to start. You need prompts and access to a strong model, not a labelled dataset, so teams can create new datasets quickly. That ease is why the practice is everywhere, sometimes without permission. By Anthropic's own account, three AI labs ran over 16 million exchanges with Claude through about 24,000 fraudulent accounts to distil it.
How is model distillation different from fine-tuning?
In LLM practice, distillation is a kind of fine-tuning. Standard fine-tuning updates a model's weights on examples written or checked by people. Distillation uses examples written by another model, and the fine-tuned models that result learn whatever it wrote. The 2026 RLHF guide covers where supervised fine-tuning, preference tuning and reinforcement learning (RL) fit in post-training. A distilled training set has no human in it unless you put one there.
What Are the Limitations of Model Distillation?
Three limits apply before any question of teacher errors.
The capacity gap
A student can be too small for its teacher. Mirzadeh et al. (AAAI 2020) found that "the student network performance degrades when the gap between student and teacher is large." The effect also shows up in complex reasoning tasks. Li et al. (2025), presented at an ICML 2025 workshop, studied models of 3 billion parameters or fewer. These models "do not consistently benefit from long chain-of-thought (CoT) reasoning" distilled from stronger reasoners. They showed better performance when shorter reasoning examples were mixed in.
Coverage
A student learns the tasks its training data covers. Outside that coverage, its reliability drops, even when it sounds as fluent as the teacher. The imitation study in the next section shows how large that gap can be.
Licensing
Some providers' terms restrict training on their outputs. OpenAI's terms of use list "Use Output to develop models that compete with OpenAI" among the things customers cannot do. Check the teacher's terms before you build a student on it.
What Does a Student Inherit in Knowledge Distillation?
Kili's webinar on fine-tuning LLMs made the basic point. If the teacher lacks expertise in a domain, the distilled data and the student will inherit that gap. Research since then shows the inheritance is broader, and harder to see, than a simple copy of the teacher's skill level.
Style travels further than facts
Gudibande et al. (ICLR 2024) fine-tuned open models on ChatGPT outputs and asked crowd workers to rate them. Raters scored the imitation models as competitive with ChatGPT. Targeted automatic evaluations told a different story. On tasks the imitation data didn't cover well, the models closed little to none of the gap to ChatGPT. The authors' summary is that imitation models are "adept at mimicking ChatGPT's style but not its factuality."
For anyone distilling a domain model, this is the first trap. A student that sounds like the teacher will pass a quick read. Whether it knows what the teacher knows is a separate question.
Confident errors on rare cases transfer, while the average improves
Lukasik et al. (Google Research, TMLR 2022) found that distillation can raise overall accuracy while hurting specific subgroups. One instance is classes with few training examples. On those classes the teacher is often wrong but confident, and "such misplaced confidence is largely transferred to the student." The headline metric goes up. The rare cases, often the ones a domain expert cares about most, get worse.
Cheng, Agarwal and Amiri (2025) report a similar pattern for spurious correlations (shortcuts that hold in the training data but not in general). Distillation generally weakened the model's robustness to them. The overall robustness number could stay stable while individual bias types shifted widely.
Traits can pass through data that doesn't mention them
Cloud et al., published in Nature in April 2026, go further. The researchers gave a teacher model a trait, such as a preference for owls or misaligned behaviour. The teacher then generated data unrelated to that trait: number sequences, code, maths reasoning. The researchers filtered out every reference to the trait. A student fine-tuned on the filtered data still acquired it. Students trained on numbers from misaligned teachers inherited misalignment. In the paper's words, "the phenomenon persists despite rigorous and validated filtering."
The finding has limits. The effect appeared only when teacher and student shared a base model, meaning both were fine-tuned from the same model. The traits were also deliberately implanted in controlled experiments. So this doesn't mean every distilled model carries hidden behaviours. It does mean that checking the data for a property doesn't guarantee the property didn't transfer.
The copy is imperfect in unpredictable ways
Stanton et al. (NeurIPS 2021) add a final complication. Students usually don't match their teachers closely, even when they have the capacity to. The authors found "a surprisingly large discrepancy between the predictive distributions of the teacher and the student." What the student inherits is a partial copy you can't read off a leaderboard.
Why Can't Student-Side Testing Catch Inherited Errors?
Each of these findings has the same shape: the inherited problem doesn't show up in the aggregate number teams usually watch. Crowd ratings look good while factuality lags. Mean accuracy rises while rare classes degrade. Overall robustness holds while specific biases move. A trait passes through data that a filter has checked.
Testing the student after training can tell you something is wrong on a slice you thought to test. It can't tell you where the error came from, and it won't flag the slices you didn't test. Retraining on the same teacher data reproduces the same errors. The information you need, which teacher outputs were wrong and why, exists only before training.
Some evidence points the other way. OpenThoughts (Guha et al., ICLR 2026) is a large study of data recipes for reasoning models. It tested many ways of automatically filtering or verifying teacher answers. It found that "none gave significant performance improvements" on its reasoning benchmarks. It also found that the best-scoring model is not necessarily the best teacher.
That result concerns automated correctness filters on benchmarks with checkable answers. It doesn't show teacher errors are harmless. It also couldn't detect the inheritance described above, because benchmark accuracy is the measure that hides it. The errors worth catching in a domain model are rarely the ones a string-match on the final answer finds.
How Do You Check a Teacher's Outputs Before Distillation?
The process below treats the teacher's outputs as a dataset to audit, the same way you would audit labels from a new annotation vendor.
[IMAGE PLACEHOLDER: model-distillation-check-before-training.webp (1600 × 900). Alt text: Teacher outputs pass an expert check before they become the training set for the student model; outputs that fail the check are fixed, dropped or re-sampled.]
1. Decide what "correct" means, with the people who know
Before sampling anything, domain experts define what a good output looks like for the student's task, and which cases are high-risk. The result is an expert-checked reference set to score the teacher against. Kili's guide to building golden datasets covers how to build and govern one.
2. Run the automatic checks you can
For verifiable outputs, filter automatically. DeepSeek's pipeline sampled several responses per prompt and kept "only the correct ones." It then dropped outputs with mixed languages, long paragraphs or code blocks. Automated verifiers have their own failure modes, which our guide to RLVR covers in detail. Treat this step as a first pass.
3. Sample deliberately and have experts review
Draw a sample of teacher outputs that focuses on rare classes, edge cases and high-risk prompts. That's where Lukasik et al. found confident errors transfer. If the teacher produced rationales, review those too. Domain experts evaluate each output against the reference definition and tag the error type.
eBay's distillation study (Lee et al., 2025) shows why the tagging step pays off. The team distilled GPT-3.5 customer-service summaries into Llama 3.1 students. First, they analysed the teacher's frequent errors. In one channel, 38% of the errors were unnecessary mentions of the customer asking for a human agent. They corrected those error types before fine-tuning. The corrected 8B student scored 4.33, against 4.14 for the same student trained on uncorrected outputs and 4.05 for the teacher. It is one task at one company, scored by an LLM judge. But finding the teacher's systematic errors first is the part that transfers.
4. Fix or drop, then measure the review itself
Correct the error types you found, or drop the affected outputs, and re-sample. Then check that the review is reliable. If two experts disagree on whether a teacher output is acceptable, the definition from step 1 needs work. If an expert misses errors on items with known answers, their verdicts on the rest are suspect.
5. Keep a record of what was checked
When a distilled model misbehaves in production, the first question is whether the behaviour came from the teacher. To answer it, you need to know which teacher outputs were reviewed, by whom, with what agreement, and what was changed.
Where Kili fits
Kili doesn't run the distillation. The teacher calls and the student training run in your own stack. Kili is where the teacher's outputs get expert review before they become training data. Teacher outputs are imported as text assets. Any automated verdict or score is uploaded as a prediction that the domain expert confirms or corrects. Consensus, enabled on a labelling step, measures how often experts agree when they label the same output. Labelling stays blind, so they can't see each other's verdicts. Low agreement tells you the definition of "correct" from step 1 isn't clear yet. Honeypot scores each expert's work against items with known answers. Enforce Step Separation, on by default, means no one reviews their own work. For teams that want to review model outputs in a dedicated interface, Kili's LLM labeling projects are in private beta; ask the Kili team for access.
Conclusion: Distillation Moves the Labelling Decision, It Doesn't Remove It
A distilled training set looks label-free, and that is its appeal. In practice the teacher wrote the labels. Every judgement it made, good or bad, is now a training example. Hinton's original insight was that the teacher's view of the wrong answers is valuable information. The research since then adds that the student absorbs that view in ways the usual metrics don't surface.
Teams that distil a frontier model into a domain model are running an annotation project, whether they planned to or not. The annotator is a model. The question that decides the student's quality is the one any annotation project asks. Who checked the labels, against what standard, and how do you know the checking was reliable? Teams that answer it before training get a student they can explain. Teams that answer it after deployment are reverse-engineering their own model.
Resources
Distillation foundations
- Distilling the Knowledge in a Neural Network – Hinton, Vinyals & Dean (2015), arXiv
- Distilling the Knowledge in a Neural Network (HTML version) – soft targets, temperature and the "mythical digit" experiment
- Knowledge Distillation: A Survey – Gou, Yu, Maybank & Tao (2021), International Journal of Computer Vision
- Sequence-Level Knowledge Distillation – Kim & Rush (2016), EMNLP
- On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes – Agarwal et al. (2024), ICLR
- Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes – Hsieh et al. (2023), Findings of ACL
Distillation in practice
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter – Sanh et al. (2019), arXiv
- Improving Streaming Automatic Speech Recognition With Non-Streaming Model Distillation On Unsupervised Data – Doutre et al. (2020), arXiv, Google
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning – DeepSeek-AI (2025), arXiv; distilled Qwen and Llama models
- Model Distillation in the API – OpenAI (2024)
- Amazon Bedrock Model Distillation – AWS product page
- Detecting and preventing distillation attacks – Anthropic (2026)
- OpenAI Terms of Use – restriction on using outputs to build competing models
- OpenThoughts: Data Recipes for Reasoning Models – Guha et al. (2025), ICLR 2026
- Targeted Error Correction in Knowledge Distillation: Small Language Models Surpass GPT – Lee et al. (2025), arXiv preprint, eBay
Limits and what students inherit
- Improved Knowledge Distillation via Teacher Assistant – Mirzadeh et al. (2020), AAAI
- Small Models Struggle to Learn from Strong Reasoners – Li et al. (2025), ICML 2025 AI for Math workshop
- The False Promise of Imitating Proprietary LLMs – Gudibande et al. (2024), ICLR
- Teacher's Pet: Understanding and Mitigating Biases in Distillation – Lukasik et al. (2022), TMLR
- Do Students Debias Like Teachers? – Cheng, Agarwal & Amiri (2025), arXiv preprint
- Language models transmit behavioural traits through hidden signals in data – Cloud et al. (2026), Nature
- Does Knowledge Distillation Really Work? – Stanton et al. (2021), NeurIPS
Kili guides
- 2026 RLHF Guide – where SFT, preference tuning and RL fit in post-training
- Webinar Recap: Techniques in Fine-Tuning LLMs – distillation and human review as data sources
- How to Build Golden Datasets for Testing, Fine-Tuning and Evaluating AI Models
- RLVR Explained: Reinforcement Learning with Verifiable Rewards – how automated verifiers get gamed
- Kili Technology
Ready to Review Your Teacher's Outputs Before You Distil?
Kili lets your domain experts review a teacher model's outputs before they become a student's training data, with consensus and honeypot checks configured in each project's workflow. Talk to the Kili team about setting up a review project.
