Preventing LLM Hallucinations at the Source: A Training Data Guide

Introduction

AI hallucinations arise when a model produces information that is factually incorrect or fabricated — entities that do not exist, relationships that do not hold, answers to questions the model has insufficient training data to answer. The standard playbook for training an extraction or generation model looks like this: collect examples of the thing you want the model to produce, label them, and fine-tune. Teams spend weeks sourcing positive examples, building annotation guidelines around them, running inter-annotator agreement studies on them. The negative side of the dataset (the spans the model must not extract, the questions it should refuse to answer, the contexts where the correct output is empty) gets treated as an afterthought. Often it isn't designed at all. It's whatever falls out of the annotation process as "not labeled."

This is a problem, because hallucination is not a failure to learn positive patterns. It is a failure to learn negative ones. A model that has seen thousands of correct entity extractions but few examples of text that looks like an entity but isn't will eagerly extract entities from thin air. A model that has never been shown a question it should refuse will always attempt an answer.

The research literature has caught up to this over the past three years. Training with explicit negative instances improves zero-shot NER by 9 F1 points. Hallucination-focused preference optimization reduces hallucination rates by 96% across five language pairs. Clinical NER models trained on carefully constructed false-positive sets achieve 50–90% false positive reduction. The evidence is consistent: what you exclude from training data matters at least as much as what you include.

This guide covers how to build those negative cases into your own training data. The approach applies to natural language processing tasks like NER, relation extraction, question answering, dialogue, and any supervised fine-tuning task where the model needs to learn when to say nothing.

Why Do Models Hallucinate When Negative Examples Are Missing?

The mechanism is straightforward. In a typical NER dataset, most tokens are labeled as non-entities by default — they're the background. But the model never sees an example of text that resembles an entity and is explicitly labeled "this is not one." So it learns to recognize surface patterns (capitalized words, domain-specific terminology, numerical expressions) without learning where those patterns should not fire.

Li et al. demonstrated this at ICLR 2021 with what they called the unlabeled entity problem. In many NER datasets, genuine entities go unannotated — sometimes because annotators missed them, sometimes because the label schema didn't cover them. The model encounters these unlabeled entities during training, treats them as negatives (they have no label), and learns the wrong lesson: that some real entities are not entities. The result is confused entity boundaries and hallucinated entity types.

Their fix was negative sampling designed to avoid exposing models to unlabeled entities. Entity sparsity in datasets provides a statistical guarantee: randomly sampled negative spans are unlikely to contain unlabeled entities simply because genuine entities are rare. When they applied this sampling strategy, the misguidance from unlabeled entities was nearly eliminated.

The broader implication is that "unlabeled" is not the same as "negative." If you want a model to learn what a negative case looks like, you have to show it negative cases that are designed to be negative — not just whatever happened to be left unlabeled.

What Makes a Good Negative Example?

Not all negatives are equal, and the distinction has measurable consequences for how models generalize.

Barton et al. published a study in Nature Machine Intelligence (2025) examining how training data composition determines generalization. Their finding cuts against intuition: models trained on negatives that are more similar to positives achieve better out-of-distribution performance, despite lower in-distribution accuracy. The binding rules the model learns change depending on the negatives it's trained against.

This is the case for hard negatives — examples that are close to the decision boundary, plausible enough to be confusing, but definitively wrong. A hard negative for a biomedical NER model isn't the word "table" appearing in a clinical note. It's "fibromyalgia-like syndrome" in a context where the patient doesn't have fibromyalgia and the phrase is a differential being ruled out. The surface features match; the label doesn't.

Li et al. (2025) built a dynamic hard negative augmentation framework for entity and relation extraction that continuously optimizes the proportion of hard negatives during training. Their system outperformed uniform negative sampling on boundary detection and complex relationship extraction — exactly the cases where hallucination risk is highest, because the model has to distinguish between entities that nearly overlap and relations that almost hold.

The clinical domain offers some of the most concrete evidence. Jiwani et al. (2025) at ConcertAI built a noise reduction model for clinical BERT NER that selected false positives based on lexical similarity with contextual irrelevance: terms that look like clinical entities in isolation but aren't entities in their specific context. They constructed labeled sets of true positives versus these near-miss false positives and used a Probability Density Map to capture what they call the "Semantic-Pull effect" in transformer embeddings. The result was a 50–90% false positive reduction across clinical NER models.

The lesson for practitioners: your negative examples should be designed to be difficult, not merely present. Random negatives teach the model easy distinctions it would probably learn anyway. Hard negatives teach it the distinctions that prevent hallucination in production.

How Should You Design Negatives for Different Task Types?

The negative-example strategy varies by task, but the underlying logic is the same: show the model a plausible input where the correct output is either empty, a refusal, or a correction.

Named entity recognition. Collect spans that share surface features with real entities but are not entities in context. Medical text: drug names used as adjectives ("aspirin-like properties"), conditions mentioned in family history but not present in the patient, abbreviations that expand differently in different specialties. Legal text: party names that appear in cited precedent rather than the current case. Financial text: numbers that are page references or footnote markers, not monetary amounts. The clinical NER data from Jiwani et al. provides a template — they tracked distributions like 6,700 true biomarker entities against 24,500 similar false entities, giving the model a realistic ratio of hard negatives to genuine entities.

In practice, this means running dedicated negative-case annotation alongside the positive labeling, not treating them as the same job. One team building a golden dataset for clinical trial extraction (entities like trial phases, arm types, outcome measures, and statistical comparisons, plus relations linking results to treatment arms) split their annotation into two parallel tracks. The first labeled positive cases: real entities and valid relations in trial publications. The second was dedicated entirely to hard negatives. Domain experts marked spans the model must not extract and contexts where the correct output is null, tagging each with a reason code explaining why the span is a negative. A trial phase mentioned in the literature review section rather than the study's own methodology. An outcome measure reported for a different treatment arm. A p-value from a secondary analysis, not the primary endpoint. The reason codes structured the negative taxonomy for the model and also let the team audit whether annotators were applying consistent logic about what counts as a false positive versus what's simply unlabeled.

Relation extraction. The same clinical trial team's work illustrates why negatives matter even more for relations than for entities. Their schema included relations like "result belongs to arm" and "outcome compared across arms," and the most dangerous false positives were relations that almost held: a statistical result that appeared in the same paragraph as a treatment arm but actually described a different comparison, or an outcome measure that was discussed in relation to a trial but came from a different study cited in the same paper. These near-miss relations are exactly the kind of plausible-but-wrong output that an extraction model will produce if it has only learned from positive examples.

Kuzman and Pollak (2024) confirmed this pattern quantitatively. Their study showed that hallucinations in synthetic training data for relation extraction cause recall drops of 19–39%, and only relevant hallucinations (plausible but incorrect facts) damaged performance. Irrelevant hallucinations had minimal impact. The negative relation examples that matter are the ones that almost hold: "Company X acquired Company Y" when Company X only invested. "Drug A treats Condition B" when Drug A was studied for Condition B but failed Phase III.

Question answering and dialogue. FaithDial, published in TACL 2022 by Dziri et al., created hallucination-free dialogue training data by editing hallucinated responses to make them faithful — including teaching models to acknowledge ignorance when the question goes beyond the provided context. Training on this data boosted hallucination critic performance by 12.8 F1 and transferred zero-shot to other datasets. The design principle: pair the hallucinated response with the faithful one, and include explicit cases where the faithful response is "I don't have enough information to answer that."

Code completion. The recent "Synthetic Hallucinations, Real Gains" paper (2026) applied this same logic to code autocomplete: using frontier models to generate plausible-but-wrong completions as hard negatives, then training on the contrast between synthetic hallucinations and ground-truth developer edits. Fine-tuning Qwen2.5-Coder-7B on a curated 100K-row subset of these pairs improved exact match by 18.8 points.

What Can Benchmark Design Teach You About Building Hard Negatives?

Two well-known benchmarks were built on the same logic that makes hard negatives effective in training data — they just apply it at evaluation time.

TruthfulQA, developed by Lin, Hilton, and Evans (ACL 2022), uses 817 adversarially crafted questions designed to exploit common misconceptions. Each question is a trap: it targets a falsehood that models are statistically likely to reproduce because the wrong answer appears frequently in their training distribution. Their finding — that larger models were less truthful — suggests that scale amplifies the pattern-matching that makes false answers attractive, rather than correcting it.

HaluEval, from Li et al. (EMNLP 2023), used a sampling-then-filtering framework to construct 35,000 hallucination evaluation examples. The process: instruct the model to generate hallucinated samples, then filter for the most plausible and difficult cases. The filtering step is the key — it ensures the negatives are genuinely hard, not trivially distinguishable.

Practitioners can borrow both methodologies for adversarial training data construction. The TruthfulQA approach suggests collecting LLM outputs on adversarial inputs and labeling the confident-but-wrong ones as explicit negatives. The HaluEval approach suggests using the model itself to generate hallucinated candidates, then curating the most plausible failures into a training set. In both cases, the hard negative works because it targets the specific boundary where the model is most likely to fail — not a random boundary, but the learned boundary where training distribution biases create systematic errors.

Why Should Training Data Teach Models to Say "I Don't Know"?

Ferrando et al. at Google DeepMind, in a paper accepted at ICLR 2025, found that language models have internal "entity recognition" directions — representations that encode whether an entity is one the model can recall facts about. These directions are causally active: they can be manipulated to make the model refuse to answer about unknown entities or, conversely, to hallucinate confidently about them.

This has a direct implication for training data design. If models have an internal mechanism for recognizing what they know, that mechanism can be strengthened or weakened by the training distribution. A dataset that never includes unknown entities — where every question has an answer and every context contains an extractable target — trains the model to suppress its own uncertainty signals.

The legal NER work by Vargas et al. (2025) illustrates this concretely. Base LLaMA-2 7B showed significant hallucinations on Spanish legal documents even at zero temperature. The model's "eagerness to find an answer" produced entity extractions from text that contained no entities of the requested type. Fine-tuning with domain-specific examples that included correct null outputs — contexts where the answer was genuinely "no entity present" — improved accuracy from 61.7% to 79.4% and substantially reduced hallucinations.

The fix is to explicitly include unanswerable inputs in training data:

• Passages that contain no entities of the target type, labeled as empty.
• Questions where the provided context does not contain the answer, labeled with a refusal.
• Dialogue turns where the correct response is to acknowledge uncertainty rather than guess.

These aren't edge cases to handle after the fact. They are a core category of training data, as important as the positive examples.

How Does Preference Optimization Scale Hard Negative Training?

For teams operating at scale, preference optimization offers the most efficient path to hallucination reduction. The idea: generate candidate outputs, classify them as faithful or hallucinated, and train the model to prefer the faithful ones.

Tang, Chatterjee, and Garg demonstrated this at NAACL 2025 with hallucination-focused preference optimization for machine translation. They created synthetic preference datasets where the "chosen" output was faithful to the source and the "rejected" output contained hallucinated content. The results: a 96% average hallucination rate reduction across five language pairs, with 89% reduction transferring zero-shot to unseen language pairs.

The scalability comes from two properties. First, generating hallucinated candidates is cheap — the model produces them naturally when prompted with adversarial or ambiguous inputs. Second, classifying outputs as hallucinated versus faithful can be partially automated using reference-based metrics or a separate critic model. The human effort concentrates on validation and edge cases rather than construction from scratch.

This approach is most practical when you already have a deployed model generating outputs. Collect the outputs, identify the hallucinated ones (through automated checks, user feedback, or expert review), pair them with corrected versions, and use the pairs for preference fine-tuning. Each iteration produces new hard negatives from the model's actual failure modes — the hardest possible negatives, because they come from the model's own learned biases.

What Does This Look Like in Practice?

Building hard negatives into your training data works best as a continuous process running alongside your annotation pipeline. Here is a concrete sequence:

Start with your model's failures. Before designing negatives from scratch, run your current model on a held-out set and collect the false positives and confident wrong answers. These are your first-generation hard negatives. They represent the specific decision boundaries where your model currently hallucinates.

Build a false-positive taxonomy. Categorize the failures by type: surface-similarity errors (the model matched on lexical features), context-ignorance errors (the model extracted an entity from the wrong context), eagerness errors (the model produced an output when the correct answer was empty). Each category needs its own class of negative examples. Some teams formalize this by tagging every negative annotation with a reason code that explains why it's negative, not just that it's negative. The reason codes double as your taxonomy and make it possible to track which failure categories the model is improving on and which are persisting across training iterations.

Set explicit ratios. The clinical NER data from Jiwani et al. offers a reference: their datasets contained roughly 3.5× more false entities than true entities for some categories. The Nature Machine Intelligence paper suggests that higher similarity between negatives and positives improves out-of-distribution performance. Neither paper prescribes a universal ratio, but the direction is clear — you need substantially more negatives than most teams currently include, and those negatives should be hard.

Include null-output examples. For every task type, create inputs where the correct output is nothing. Passages with no entities. Questions the context can't answer. Dialogue turns where the model should say it doesn't know. These should constitute at least 10–15% of your training set.

Iterate with preference pairs. Once the model is deployed, use its production outputs to generate new preference data. Hallucinated outputs become "rejected" examples; corrected outputs become "chosen" examples. This creates a feedback loop where each deployment cycle produces harder negatives tailored to the model's current weaknesses.

The teams that sustain this loop over time are the ones whose annotation platforms support it operationally: honeypot-based quality tracking to catch annotator drift on negative cases, consensus metrics to validate that hard negatives are genuinely ambiguous (not just mislabeled), and configurable review workflows that let domain experts audit the negative examples separately from the positives. Without that infrastructure, the negative-example pipeline decays within a few iterations as false-positive taxonomies go stale and ratio targets slip.

Where Does Hard Negative Training Fit Among Other Prevention Methods?

Training data design is one of several intervention points for hallucination prevention. The methods are complementary, and most production systems use more than one.

Inference-time decoding strategies modify token generation without changing model weights. Context-Aware Decoding amplifies the influence of provided context during generation, overriding the model's prior knowledge when it contradicts the input. DoLa (Decoding by Contrasting Layers) compares token probabilities across transformer layers to detect and suppress likely hallucinations at generation time. These methods are attractive because they require no retraining, but they add latency and can only suppress hallucinations the model is already uncertain about. They don't fix a model that confidently produces incorrect information.

Retrieval-augmented generation (RAG) grounds the model's output in retrieved evidence. The most effective RAG systems add span-level attribution, checking each generated claim against retrieved passages and flagging unsupported statements. RAG reduces hallucinations substantially for factual queries, but introduces its own failure modes: outdated or irrelevant retrieval, misinterpretation of retrieved context, and hallucinated synthesis across multiple sources. A model that hallucinates entities in extraction tasks won't be fixed by better retrieval.

Prompting techniques like chain-of-thought reasoning and self-verification reduce hallucinations by forcing the model to show its work. A 2025 survey in Frontiers in AI found that CoT prompting reduced medical hallucinations in up to 86.4% of evaluated comparisons. Self-consistency methods sample multiple outputs and select the most internally consistent one. These approaches are cheap to implement but are fundamentally prompt-dependent. They work for question answering and reasoning tasks; they don't apply to supervised extraction or classification models.

RLHF and DPO alignment train the model to prefer factual outputs using human preference signals. RLHF collects hallucinated and non-hallucinated responses, trains a reward model, and fine-tunes the LLM using reinforcement learning. DPO simplifies this by optimizing directly on preference pairs. Both methods have shown strong results, but they share a vulnerability: preference labels can be misaligned with factuality when annotators prefer fluent or confident responses over correct ones. The quality of the preference data determines whether alignment training reduces hallucinations or inadvertently reinforces them.

Runtime guardrails and detection classify outputs after generation, using either external classifiers or probes on the model's own activations. These catch hallucinations before they reach users, but they're reactive. They don't improve the model — they filter its output.

Each method addresses hallucination at a different point in the pipeline: architecture, training, alignment, inference, or post-generation filtering. But training data design is the most upstream intervention, and the one with the strongest empirical evidence for lasting improvement. Decoding strategies and guardrails manage symptoms at inference time. RAG adds external knowledge but doesn't change what the model has learned. RLHF and DPO improve alignment but depend on the quality of the preference data they're trained on. Training data composition changes what the model knows, including what it knows not to do.

This is also the intervention that maps most directly to an operational workflow. Building hard negatives requires the same infrastructure that teams already use for annotation: parallel annotation jobs (positive and negative tracks running simultaneously on the same corpus), quality metrics that track annotator performance on negative cases separately from positive ones, and domain expert review for validating whether a hard negative is genuinely ambiguous or mislabeled. Kili Technology is built around this kind of workflow. Teams running hard negative annotation on Kili use honeypot quality metrics to catch annotator drift on negative cases, multi-step review to separate domain expert audit from initial labeling, and consensus measurement to validate the hardest negatives. For regulated domains where realistic hard negatives require sensitive documents (clinical trials, legal filings, financial data), the platform's project-level access controls and cloud storage integration restrictions keep sensitive data scoped to the projects that need it.

Conclusion

Most hallucination prevention methods intervene after the model has already learned the wrong patterns: decoding strategies suppress uncertain tokens, RAG grounds outputs in retrieved evidence, guardrails filter the worst mistakes before they reach users. These are useful, and most production systems should use them. But they are compensating for a training data problem.

The training data intervention is upstream. It changes what the model learns, including when to stop producing output. A well-designed hard negative tells the model where its pattern-matching should stop firing: the near-miss entity, the unanswerable question, the fluent output built on fabricated facts. Without those constraints, the model defaults to a training distribution that rewards production over restraint.

The research covered in this guide points consistently in the same direction: negative-case design is a first-class data engineering problem. It requires dedicated annotation jobs, explicit false-positive taxonomies with reason codes, careful control of negative-to-positive ratios, and preference optimization loops that use the model's own failures as training signal. The teams getting this right treat hard negatives as core training data, not an afterthought. The result is a model that knows when to stop.

Resources

Training Data Composition and Negative Examples

• Rethinking Negative Instances for Generative Named Entity Recognition (Ding et al., 2024) – Negative instances improve zero-shot NER by 9 F1 points
  • https://arxiv.org/abs/2402.16602
• Empirical Analysis of Unlabeled Entity Problem in NER (Li et al., ICLR 2021) – Unlabeled entities misguide NER models; negative sampling fixes it
  • https://arxiv.org/abs/2012.05426
• Training Data Composition Determines ML Generalization (Barton et al., Nature Machine Intelligence, 2025) – Negative data composition changes learned rules and OOD performance
  • https://www.nature.com/articles/s42256-025-01089-5
• Dynamic Hard Negative Augmentation for Entity and Relation Extraction (Li et al., 2025) – Continuous hard negative optimization outperforms uniform sampling
  • https://www.sciencedirect.com/science/article/abs/pii/S0952197625022195

Hallucination Detection and Reduction

• Noise Reduction in BERT NER for Clinical Entity Extraction (Jiwani et al., 2025) – 50–90% false positive reduction via lexically similar negative sets
  • https://arxiv.org/abs/2603.00022
• Effects of Hallucinations in Synthetic Training Data for Relation Extraction (Kuzman & Pollak, 2024) – Relevant hallucinations cause 19–39% recall drops
  • https://arxiv.org/abs/2410.08393
• Hallucination-Focused Preference Optimization (Tang et al., NAACL 2025) – 96% hallucination reduction via synthetic preference pairs
  • https://arxiv.org/abs/2501.17295
• Fine-Tuning LLaMA for Legal Entity Extraction (Vargas et al., 2025) – Null-output training reduces legal NER hallucinations
  • https://arxiv.org/abs/2506.08827

Benchmarks and Evaluation Methodologies

• TruthfulQA: Measuring How Models Mimic Human Falsehoods (Lin et al., ACL 2022) – 817 adversarial questions exploiting training distribution biases
  • https://arxiv.org/abs/2109.07958
• HaluEval: A Large-Scale Hallucination Evaluation Benchmark (Li et al., EMNLP 2023) – Sampling-then-filtering for 35,000 hallucination evaluation examples
  • https://arxiv.org/abs/2305.11747
• FaithDial: A Faithful Benchmark for Information-Seeking Dialogue (Dziri et al., TACL 2022) – Faithful dialogue training including explicit ignorance acknowledgment
  • https://arxiv.org/abs/2204.10757

Model Self-Knowledge

• Do I Know This Entity? Knowledge Awareness and Hallucinations in LMs (Ferrando et al., ICLR 2025) – Models have internal entity recognition directions that can steer refusal or hallucination
  • https://arxiv.org/abs/2411.14257

Platform and Tools

• Kili Technology – Annotation platform with honeypot quality tracking, multi-step review workflows, and per-labeler quality metrics for hard negative annotation
  • https://kili-technology.com

Other Hallucination Prevention Methods

• Correction with Backtracking Reduces Hallucination in Summarization (Context-Aware Decoding) – Contrastive decoding overrides model priors when they contradict provided context
  • https://arxiv.org/pdf/2310.16176
• Attribution Techniques for Mitigating Hallucinated Information in RAG Systems: A Survey – Span-level attribution for verifying generated claims against retrieved evidence
  • https://arxiv.org/pdf/2601.19927
• Survey and Analysis of Hallucinations in LLMs (Frontiers in AI, 2025) – Comprehensive taxonomy of hallucination causes and mitigation across prompting, decoding, and training
  • https://www.frontiersin.org/journals/artificial-intelligence/articles/10.3389/frai.2025.1622292/full
• Mask-DPO: Generalizable Fine-grained Factuality Alignment of LLMs – Fine-grained preference optimization where preference labels can be misaligned with factuality
  • https://arxiv.org/pdf/2503.02846

‍

Frequently Asked Questions

What are hard negatives in AI training data?

Hard negatives are training examples that closely resemble positive cases but are definitively wrong. In NER, a hard negative might be a medical term that looks like a clinical entity but isn't one in context — "fibromyalgia-like syndrome" used as a differential being ruled out, not an actual diagnosis. Research in Nature Machine Intelligence (2025) shows that models trained on hard negatives generalize better out-of-distribution than those trained on random negatives.

Why do large language models hallucinate?

LLMs hallucinate because their training data overwhelmingly contains positive examples — correct outputs the model should produce — without enough explicit negative examples showing what not to produce. A model that has seen thousands of correct entity extractions but few examples of text that resembles an entity but isn't one will extract entities from thin air. Research at ICLR 2021 showed that unlabeled entities in NER datasets actively misguide models by teaching them to treat real entities as non-entities.

How do negative examples reduce AI hallucinations?

Negative examples teach the model where its pattern-matching should stop firing. Adding explicit negatives to generative NER training data improves zero-shot F1 by 9 points. Clinical NER models trained on carefully constructed false-positive sets achieve 50–90% false positive reduction. The key is that negatives must be designed to be difficult — plausible enough to confuse the model — rather than trivially distinguishable from real examples.

What is null-output training for LLMs?

Null-output training means including inputs in your training data where the correct output is nothing: passages with no entities to extract, questions the context can't answer, or dialogue turns where the model should say "I don't know." Vargas et al. (2025) showed that fine-tuning LLaMA with null-output examples improved legal NER accuracy from 61.7% to 79.4% and substantially reduced hallucinations caused by the model's eagerness to always produce an answer.

How does preference optimization scale hallucination reduction?

Preference optimization creates training pairs where the model learns to prefer faithful outputs over hallucinated ones. Tang et al. (NAACL 2025) demonstrated a 96% hallucination rate reduction across five language pairs using synthetic preference datasets, with 89% reduction transferring zero-shot to unseen languages. The approach scales well because hallucinated candidates are cheap to generate (the model produces them naturally) and classification can be partially automated.

What infrastructure do you need for hard negative annotation?

Hard negative annotation requires parallel annotation tracks (positive and negative labeling running on the same corpus), quality metrics that track annotator performance on negative cases separately, and domain expert review for validating ambiguous negatives. Kili Technology supports this workflow with honeypot quality metrics for catching annotator drift on negative cases, multi-step review workflows for separating expert audit from initial labeling, and consensus measurement for validating the hardest negatives. For regulated domains, project-level access controls keep sensitive source documents scoped to the projects that need them.

How does training data design compare to RAG for preventing hallucinations?

RAG grounds model output in retrieved evidence and reduces hallucinations for factual queries, but it doesn't change what the model has learned — a model that hallucinates entities in extraction tasks won't be fixed by better retrieval. Training data design is the most upstream intervention: it changes the model's internal representations, including when to suppress output. The two methods are complementary, and most production systems benefit from both.

What ratio of negative to positive examples should training data include?

There is no universal ratio, but the research points toward substantially more negatives than most teams currently include. Clinical NER datasets from Jiwani et al. (2025) contained roughly 3.5× more false entities than true entities for some categories. Null-output examples (inputs where the correct answer is empty) should constitute at least 10–15% of the training set. The negatives should also be hard — similar to positives — since higher similarity improves out-of-distribution performance.

‍