Resources
/
AI Evaluation
Guides, data stories and answers on evaluating AI with the people who can tell a right answer from a plausible one. Build golden datasets and benchmarks on your own domain, learn where LLM-as-a-judge needs a second opinion, and catch regressions before a release ships.
01
02
Short answers to the questions teams ask us most in this area.
Choose the assets your model will be judged on, have domain experts label them against a fixed ontology, and measure their agreement before you freeze the set. Include deliberately hard and negative cases, so the set tests failure modes and not only easy wins. In Kili, consensus sends the same asset to several experts and scores where they diverge, and review steps resolve the disagreements. The validated labels can then be uploaded as honeypots, so future labelers are measured against the same reference.
LLM-as-a-judge uses one language model to grade another model's outputs, which makes evaluation cheap to run at volume, but it is only reliable once you know it agrees with the experts it replaces. In Kili, the judge's verdicts can be imported as predictions tagged with the judge's model name, and specialists grade the same outputs. Both sets of labels can then be retrieved through the Python SDK and compared. The expert-graded set stays available for re-checking the judge after every prompt or model change.
To evaluate a RAG system, judge retrieval and generation separately, because a retrieval-augmented answer can fail at either step. Retrieval asks whether the passages pulled from your documents were relevant and complete. Generation asks whether the answer is faithful to those passages and addresses the question. In Kili, the retrieved context, the question and the answer can sit on one asset, with a separate classification job for each criterion, so a low score points at the step that failed.
Send the same outputs to more than one evaluator and measure the gap. With consensus on the evaluation step, Kili scores agreement per class and per evaluator, so a criterion that experts read differently shows up as a low score early. Reviewers can then compare all the judgments submitted on an asset in the Explore interface and settle the disagreement.
Report
Newsletter
New guides and data stories, by email
New posts from the AI Evaluation, Post-Training and Data Labeling hubs, sent when they go live.