AI Summary
- Score each field by data type; count omitted and invented values separately.
- Missed data was the top error in a 9,341-element study, with or without an LLM.
- Reviewers caught 82% of spelling errors in pre-fills, 31% of rule violations.
- Calibrated routing still sends 20% to 51% of fields to people.
Introduction
In a study run inside six systematic reviews, a large language model (LLM) extracted 9,341 data elements from 63 research papers, and a human checked its work. A second team extracted the same papers by hand. The LLM-plus-human team was slightly more accurate (91.0% against 89.0%) and saved a median of 41 minutes per paper. Yet the two teams agreed on only 77.2% of the data. And both made the same most common error: information that never made it into the record.
That last finding matters most for teams using AI document extraction to replace manual data entry on insurance claims, policy schedules, loan files or know-your-customer (KYC) packs. Manual document processing has its own human error: the hand-extraction team in that study got 11.0% of data elements wrong. But when a human reviewer becomes the control on an AI system, the control works on what appears on screen. A wrong value can be spotted. A value the model never extracted gives the reviewer nothing to look at.
This guide covers evaluating AI document extraction when domain experts do the reviewing, with insurance document processing as the running example. It starts with how extraction works and what to score. Then it looks at where expert reviewers miss errors and why many disagreements come from the field specification rather than the model. The last sections cover how much work still needs a person, what insurance regulators expect from the review layer, and how to run the whole setup on Kili's PDF and text projects.
What Is AI Document Extraction, and How Does It Work?
AI document extraction is the use of machine learning models, today usually an LLM, to read a document and return the values a schema asks for as structured JSON or another structured format. Those values are fields such as a policy number, an insured amount or a claim date. When an LLM does the reading, teams also call it LLM data extraction, AI data extraction or document data extraction. Vendors sell it as document AI or document intelligence. It is the extraction step of intelligent document processing (IDP). It turns unstructured data into structured datasets for business systems such as claims platforms, ERPs (enterprise resource planning) and CRMs (customer relationship management). The evaluation problem is the same whether you extract data from one claim form or extract structured data from PDF files by the million.
The inputs vary. Some are structured documents and structured forms with fixed fields, such as tax forms or claim forms. Others are unstructured files: correspondence, contracts, adjuster notes and medical records. Most document processing workflows mix structured and unstructured data across document formats, from native PDFs to scanned documents and phone photos. Documents vary even within one type when they come from different brokers or hospitals.
How AI document extraction works, stage by stage
- Read the text. Native PDFs carry their document content in a text layer. Scanned documents need optical character recognition (OCR), or intelligent character recognition (ICR) for handwriting. Multimodal models can also read the page image directly. The evidence points both ways. SAP and Stanford researchers found that image-only input to strong models matched or beat OCR pipelines on about 1,000 business documents. An Amazon study of transcription faithfulness found the reverse when pulling product identifiers from 150 real documents: OCR-first pipelines beat direct vision-language extraction, with a median F1 of 93.5% against 68.6%. (F1 combines precision, the share of extracted values that are correct, with recall, the share of true values that were found.)
- Recover the layout. Layout understanding rebuilds the document structure and turns the page into structured text. That covers reading order, key-value pairs, complex tables and where each value sits in multi-page documents. Preserving structure matters because complex layouts produce a distinctive error, a value read from the neighbouring table row.
- Classify and extract. Many pipelines first classify documents by type, then apply that type's schema. Forms get key value pair extraction, invoice processing and receipt processing pull out line items, and contracts get clause extraction. Pipelines also extract metadata such as sender, date and language. Older systems relied on predefined rules and a template per document layout. Machine learning and natural language processing, first in custom models and now in general-purpose LLMs, handle document variations and unstructured documents without a template for each. That flexibility is also why their errors are harder to predict.
- Validate and route. The pipeline checks outputs against business rules (formats, checksums, totals). It then writes them to downstream systems or sends them to people through exception handling.
Agentic document extraction puts an LLM agent in charge of these stages. In AgenticIE, a planner chooses tools at run time, such as OCR, language detection, extraction, verification and translation. It returns only outputs that pass a verification step. On 80 multilingual documents in German and English, the agent produced valid JSON every time, against as low as 96.4% for a single GPT-4o call. It also used 38.69 million tokens against 3.21 million and cost about $313 against $18. The authors position it as an assistive tool whose outputs qualified professionals should verify before critical use.
Why Is AI Document Extraction Hard to Evaluate?
Because well-formed output and correct output are different things. The AOE benchmark uses complex documents from law, finance and academia. On its financial tasks, one model produced parsable tables 97% of the time but got only 0.71% of the cells right. The authors' verdict: "Simply producing 'table-like' output is insufficient."
Schema size adds its own failure mode. ExtractBench, from Contextual AI, pairs 35 PDFs with JSON schemas and 12,867 gold-labelled fields. On its 369-field financial reporting schema, no frontier model tested produced valid LLM structured output at all. Where output was valid across the benchmark, field accuracy averaged 72.9%, and the authors note that valid outputs skew towards the easier cases. The longer the schema, the more often the output breaks.
Layout variety is the third problem. Google's VRDU benchmark found that "generalizing to new document templates is still very challenging." One model scored 90.51 micro-F1 on registration forms whose templates it had seen in training, and 77.29 on unseen templates. An insurer receiving claims forms from hundreds of brokers lives in the unseen-template case.
How Should You Score AI Document Extraction?
Score at the level of the field, against a reference that experts produced, with a match rule chosen for each field's data type. One accuracy figure per document hides where the model fails and which failures cost money.
Treat the schema as the specification
ExtractBench calls the schema "an executable specification": every field declares the metric used to score it, from exact string match to numeric tolerance to a semantic judgement. That is the right starting point for document review. Before anyone looks at model output, the team agrees, field by field, what counts as correct.
Pick a match rule per field type
Exact string comparison punishes harmless differences and misses harmful ones. VRDU uses type-aware matching instead: prices are converted to numbers before comparison, and dates are parsed as dates. DocVQA scores answers with a normalised edit distance (ANLS) rather than exact match "since no OCR is perfect." A one-character scan error then isn't scored like a wrong answer.
Amounts and identifiers get no tolerance in this table on purpose. A sum insured that is "close" is wrong.
Score the reading step separately when you can. Character error rate (CER) and word error rate (WER) count the substitutions, deletions and insertions needed to turn the transcription into the reference. Both divide that count by the length of the reference. The Amazon study also found that general-purpose vision-language models sometimes rewrite what they read. When the researchers scrambled letters in words, one small model rewrote 27.35% of the altered words instead of transcribing them as printed. Tesseract, a traditional OCR engine, rewrote none. Identifiers with unusual strings are worth testing for this.
Keep omissions and invented values apart
ExtractBench gives every field three states: present, null (the document has no value) and missing (the model returned nothing). That separates two errors with different consequences. A missed value means lost information. And, in the paper's words, "When gold is null but predicted has a value, the error is hallucination." An invented deductible on a claim is a different risk from a deductible the model failed to find. The two need separate counts.

Report per field, weighted by consequence
Averages flatter extraction systems. A University of Kentucky study extracted clinical features from medical records. Accuracy for several fields sat above 0.8 while F1 was far lower. Most patients simply didn't have the condition, so correct "no" answers padded the accuracy score. Insurance schemas have the same shape: most claims have no subrogation clause or prior loss. Report precision and recall per field, and give the fields that drive a payment or a coverage decision their own line.
Where Do Expert Reviewers Miss LLM Extraction Errors?
Reviewers miss what isn't shown and what looks plausible. Both are predictable, so both can be designed for.
Omissions slip past
In the preprint version of the systematic review study, missed data accounted for 5.5% of elements in the LLM-assisted arm and 6.2% in the human arm. Fabricated data, the LLM hallucination that the human verifier didn't correct, came to 0.8% against 0.5%. The authors found that most of the LLM-assisted errors were omissions that both the model and its human checker overlooked.
Pre-fills anchor the reviewer
Bias in the Loop is a randomised experiment in which 2,784 annotators checked AI pre-annotations of figures from corporate report tables. It measured which planted errors people caught. Spelling errors were caught 82% of the time and wrong-cell errors about 77%. Errors that could only be spotted by knowing the annotation rules were caught 31% of the time. Annotators with favourable views of AI undercorrected more, a pattern the authors call "dangerous overreliance". The participants were crowdworkers, not underwriters, but the ranking is the useful part: surface errors get caught, rule errors don't.
NIST's Generative AI Profile names both halves of the problem. It defines confabulation as "The production of confidently stated but erroneous or false content." It also lists "automation bias, or excessive deference to automated systems" among the risks of human-AI configurations.
What to change in the review design
Four changes target these failures directly:
- Show every extracted value at its location in the source. Google's LangExtract maps each extraction to exact character offsets, which makes it "much easier to evaluate and verify the extracted information." On a PDF, that means the value is highlighted on the page.
- Make empty fields part of the checklist. A reviewer who only confirms filled values will never find an omission.
- Write the business rules into the review guideline and test reviewers on them, since rule errors are the ones people miss.
- Put a second, independent reviewer on a sample, so the first reviewer's misses become measurable.
Why Do So Many Disagreements Trace Back to the Field Specification?
Because the specification decides what "correct" means, and most teams write it once and never revisit it. When reviewers, models and reference labels disagree, the cause is often an unclear rule rather than a careless person or a weak model.
EviStreams, a human-in-the-loop extraction system from the University of Pennsylvania, classified 1,003 disagreements between model output and the reference. 68% were methodological, meaning problems with the ground truth, extraction conventions or scoring. Only 30% were genuine model errors. Richer field specifications improved results in 40 of 51 model and form comparisons. The authors conclude that "extraction quality is shaped far more by the field specification" than by the choice of model. The Kentucky team found the same thing from the other side: some of the model's apparent false positives were true findings that their own expert annotators had missed.
The pattern holds in insurance data. A study funded by the Casualty Actuarial Society used an LLM to extract actuarial variables from claims files and had two clinical pharmacists score the output. Their weighted kappa, an agreement score corrected for chance where 1 means perfect agreement, was 0.53. They diverged on 19% of assessments, and about 60% of those divergences were boundary cases between adjacent severity levels. That is a guideline problem: two qualified experts reading the same file need a sharper definition of where "moderate" ends. (The study used synthetic claims and a small sample, so treat the numbers as an illustration.)
The practical consequences:
- The expert owns the field specification, including examples of edge cases and what to do when a document contradicts itself.
- Disagreements get adjudicated, and each adjudication either fixes a label or changes the specification.
- Teams measure reviewer agreement before using it to judge the model. Our guide to inter-rater reliability covers which statistic to use and what threshold to set.
Expert labels cost real effort. CUAD is a contract review dataset of 510 contracts and 41 clause types. Its law student annotators went through 70 to 100 hours of training, and "Each annotation was verified by three additional annotators." That is the level of rigour a reference set for regulated documents needs, applied to a smaller sample.
How Do You Build a Reference Set for LLM Document Extraction?
A reference set is a sample of documents whose fields experts have labelled and adjudicated, used to score both the model and the reviewers. Three rules keep it honest.
Sample by document type and source template, in proportion to production and with extra weight on the rare types that drive losses. VRDU's unseen-template drop is the reason: a reference set drawn from three familiar broker templates will overstate accuracy on the fourth. Include multilingual documents if production has them, and score each language separately.
Have experts label the reference documents from scratch, without the AI output on screen. The Bias in the Loop results show what pre-fills do to judgement, and a reference set built by correcting the model inherits the model's blind spots.
Double-label and adjudicate, then version the set alongside the specification. When the specification changes, the affected labels change with it. Our golden datasets guide covers sizing and maintenance.
Once the reference set exists, mix some of its documents into the live review queue. Reviewers then get scored on known answers as part of their normal queue.
How Much AI Document Extraction Still Needs Human Review?
More than the model's own confidence suggests, and less than all of it, if confidence is measured properly. Published systems that calibrate per-field confidence still send between a fifth and a half of fields to people.
The model's native signals are weak. ExtractConf is a confidence engine for invoice data extraction. In its tests, the model's mean token log-probability separated correct from incorrect fields with a ROC AUC of only 0.705, where 0.5 is a coin flip and 1.0 is perfect. As the author puts it, a model "confidently transcribing OCR noise produces high log-probabilities for a wrong answer." A 40-feature model combining OCR confidence, agreement between two extraction calls, layout and image sharpness reached 0.928. At 80% coverage, the auto-accepted fields were 99.1% correct, against a 73.3% base rate. The other 20% went to review.
A team at OCBC, the Singapore bank, reports a similar picture for straight-through processing of financial documents, where fields are approved with no human review. There, a single wrongly approved value "is a critical financial and compliance error." Their score combines three signals: what the model read, where the value sits on the page, and checks against format rules, checksums and a second model. With a gate set to keep error among auto-approved fields at or below 10%, the full pipeline auto-approved 49% to 72% of fields. The vision-language models' native confidence signals cleared only 0.1% to 7.0% at the same target.
Document automation runs on this split. Workflow automation handles the accepted tier, and exception handling sends the rest to people. The share sent to people decides how many reviewers a given document volume needs. A working plan for teams that process documents at enterprise scale:
- Route fields, not whole documents, by a calibrated score, and set the threshold from the error rate the business accepts on each field.
- Send every field in the high-consequence group to review regardless of score.
- Keep a random audit sample of auto-accepted fields, reviewed by experts, so the accepted tier has a measured error rate. Continuous monitoring of that sample shows where the model is weak.
- Recalibrate when a new document source or template arrives, since calibration learned on one set of layouts won't transfer cleanly.
The SAP and Stanford study lists what the audit sample will turn up. Expect confused characters, values taken from the neighbouring table row, misread units, and fields missed when the value has no printed label. Each error type maps to a review rule. Our guide to LLM error analysis shows how to turn audit findings into a taxonomy.
What Do Insurance Regulators Expect From the Review Layer?
In the US, regulators expect validation, testing and documentation of AI outputs, with controls scaled to how much people are involved in the final decision. The National Association of Insurance Commissioners (NAIC) adopted its Model Bulletin on insurers' use of AI in December 2023. It asks insurers to cover "Validating, testing, and retesting as necessary to assess the generalization of AI System outputs." It also covers data quality, documentation, and due diligence on third-party AI providers, including audit rights. Its definition of an AI system covers generative AI. Among the factors for calibrating controls, it lists "the extent to which humans are involved in the final decision-making process." The NAIC's adoption map lists 30 jurisdictions that had adopted the bulletin as of 13 November 2026.
In Europe, EIOPA, the EU's insurance and pensions supervisor, published an Opinion on AI governance and risk management in August 2025. It explains how existing insurance law applies to AI systems outside the AI Act's high-risk categories. It lists human oversight, data governance and record-keeping among the elements of the framework, and states that it "does not set new requirements." Privacy law applies too. Claims files carry personal data, including medical records, which brings GDPR into scope in Europe and HIPAA for US health insurers.
Neither text prescribes a review design. Both expect the insurer to show that extraction outputs were tested, that errors are known and that someone accountable checked them. A claims consultant quoted by Insurance Business describes the same division of labour for AI claims processing, calling claims AI an "evidence workbench, not an autonomous claims authority."
Vendor pages for AI-powered extraction tools tend to promise high accuracy, enterprise grade security and every document format. The bulletin's third-party due diligence gives buyers a sharper checklist:
- How was accuracy measured, per field and on whose documents?
- Where are documents processed and stored?
- What encryption and access controls apply?
- Does the tool's API write cleanly into your business processes?
An AI document review tool judged on extraction accuracy alone leaves the rest of that list unanswered.
A well-run expert review layer produces that evidence as a by-product: per-field error rates from the reference set, reviewer accuracy on known answers, and a record of which values were corrected. If the LLM comes from a vendor, the same numbers feed the third-party due diligence the bulletin describes.
How Do You Run Expert Review of AI Document Extraction in Kili?
On a PDF or text project, you import the model's output as predictions, and experts correct it in a multi-step workflow. Agreement and reference checks measure the experts while they work. Kili's LLM labeling projects are in private beta and aren't needed for this route.

Set up the project
For AI PDF data extraction, create a PDF project, or a text project if documents arrive as text. Then add a named entity recognition job (NER, which tags spans of text with a class) with one class per schema field, such as policy number, insured amount or date of loss. NER works on text and PDF assets. Document-level judgements, such as document type, can sit in a classification job on the same asset. For scanned pages with no text layer, experts can capture fields as bounding boxes with Kili's built-in OCR. It is a transcription sub-job that fills in the text under each box. OCR runs in the annotator's browser, so documents aren't sent to a server for it. Since the 23 September release, admins can set one or more OCR languages per project, and labelers can change the language per document. The same release made the Polygon tool available in PDF projects, listed in the changelog only so far.
Import the model's extractions as predictions
Convert the model's JSON into Kili's format and import it as predictions, tagged with a model name. PDF entities carry their text (content), their position on the page and their page numbers, as described in the export format reference. Predictions show with a dashed outline and switch to a solid one when an expert modifies them, so reviewers can tell model values from human ones. The 4 November release added an Imports page in project settings for uploading PDF and text prediction files and mapping their classes. Predictions are matched to documents by external ID, so the model's output file needs the same IDs as the uploaded PDFs.
When a PDF's text layer doesn't match what is printed, the 21 October release lets the expert double-click a named entity to correct its text. That feature is listed in the changelog only; the NER docs page doesn't describe it yet.
Configure review and quality checks
In the workflow, add one or more review steps after labeling, each with assigned reviewers and its own sampling rate. Enforce Step Separation, on by default, stops anyone from reviewing their own corrections. A reviewer who finds a wrong or missing value can add an issue to the specific entity and send the asset back to the original labeler, where it gets top priority.
Two checks measure the experts. Consensus sends a set percentage of documents to several experts working blind, and for NER it scores the overlap of their entities. Honeypot scores each expert against reference labels on one workflow step, which is where the reference set from earlier goes. Both are documented in Kili's calculation rules.
Score the model
Kili's quality metrics score the people. To score the model, export the latest label for each asset. Compare it field by field with the model output you imported, using the match rules your team agreed on. Kili stores model predictions, expert labels and reviews as separate label types, each with its author and date. The export carries each entity's text and page, so you can count omissions, invented values and wrong values per field.
Conclusion: The Reviewer Is Part of the Measurement
An expert's corrections are often treated as the model's error rate. That holds only if the expert is measured too. The systematic review study showed a human verifier missing the model's omissions. The crowdworker experiment showed reviewers catching typos and missing rule violations. The claims study showed two qualified pharmacists disagreeing on one case in five. Every one of those failures would have been counted as model accuracy.
As extraction models improve, the errors they leave behind are the ones a quick read misses: absent values, plausible numbers from the wrong row, edge cases the specification never covered. Expert time moves accordingly, from typing values to owning the specification, labelling the reference set and reviewing the audit sample. That makes document extraction one more case of LLM evaluation, with domain experts as the judges. For insurers and banks, the same work is what a regulator or an auditor will ask to see.
Resources
Studies of LLM-Assisted Extraction With Human Review
- Gartlehner et al. (2025), Artificial Intelligence–Assisted Data Extraction With a Large Language Model: A Study Within Reviews – Annals of Internal Medicine; LLM plus human verifier vs human-only extraction
- Gartlehner et al. (2025), AI-Assisted Data Extraction with a Large Language Model: A Study Within Reviews – medRxiv preprint with the error-type breakdown
- Kosuri et al. (2026), EviStreams: Human-in-the-Loop AI Data Extraction for Systematic Reviews in Medicine – arXiv preprint; source-grounded review and disagreement analysis
- Klusty et al. (2025), Leveraging LLMs for Structured Data Extraction from Unstructured Patient Records – arXiv preprint; expert gold set for EHR extraction
- Lieberthal et al. (2026), Leveraging LLMs for Unstructured Claims Data Analysis – arXiv preprint funded by the Casualty Actuarial Society; synthetic claims, clinician reviewers
Extraction Pipelines, Benchmarks and Metrics
- Ferguson et al. (2026), ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction – arXiv preprint (Contextual AI)
- Wang et al. (2023), VRDU: A Benchmark for Visually-rich Document Understanding – KDD 2023 (Google)
- Mathew, Karatzas & Jawahar (2021), DocVQA: A Dataset for VQA on Document Images – introduces ANLS scoring for document answers
- Zhong et al. (2025), Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction – arXiv preprint (AOE benchmark)
- Shen et al. (2026), OCR or Not? Rethinking Document Information Extraction in the MLLMs Era – arXiv preprint (SAP, Stanford)
- Lee et al. (2026), Do VLMs Read or Rewrite? On Transcription Faithfulness in Vision-Language Models – arXiv preprint (Amazon); CER/WER and rewriting
- Colakoglu, Solmaz & Fürst (2025), AgenticIE: An Adaptive Agent for Information Extraction from Complex Regulatory Documents – arXiv preprint
- Hendrycks et al. (2021), CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review – NeurIPS 2021
Reviewer Behaviour, Confidence and Routing
- Beck et al. (2025), Bias in the Loop: How Humans Evaluate AI-Generated Suggestions – arXiv preprint; randomised experiment on correcting AI pre-annotations
- Kumar (2026), Beyond Logprobs: A Multi-Signal Confidence Engine for LLM-Based Document Field Extraction – arXiv preprint; extended version of a RobustifAI @ IJCAI-ECAI 2026 paper
- Jin et al. (2026), Perception, Layout, and Validation: Calibrated Confidence for Reliable Straight-Through Processing of Financial Documents – arXiv preprint (OCBC)
- Goel & Kiraly (2025), Introducing LangExtract – Google Developers Blog; source-grounded extraction
Regulation and Governance
- NAIC (2023), Model Bulletin: Use of Artificial Intelligence Systems by Insurers – adopted 4 December 2023
- NAIC, Implementation of the AI Model Bulletin – adoption map
- EIOPA (2025), Opinion on AI Governance and Risk Management – announcement page
- NIST (2024), AI 600-1: Generative Artificial Intelligence Profile – confabulation and automation bias
- Green (2026), Claims AI's Safest First Win: Evidence Gathering – Insurance Business
Kili Documentation
- Named entity recognition
- Optical character recognition
- Importing labels and predictions
- Export format: object, entity detection and relation
- Workflow configuration
- Handling questions and issues
- Consensus overview
- Honeypot overview
- Calculation rules for quality metrics
- Changelog, 23 September 2026 (v26.2.6)
- Changelog, 21 October 2026 (v26.2.13)
- Changelog, 4 November 2026 (v26.2.15)
Further Reading on Kili
- Inter-Rater Reliability for AI Evaluation: Agreement, Gold Answers and Thresholds
- How to Build Golden Datasets for Testing, Fine-tuning, and Evaluating AI Models
- LLM Error Analysis
Ready to Put Expert Review on Your AI Document Extraction?
Kili is the platform for expert-in-the-loop AI evaluation and post-training. It runs expert correction, multi-step review, consensus and honeypot checks on PDF and text projects, in the cloud or on your own infrastructure. Talk to the Kili team about setting up a review workflow for your documents.
.png)