Ultimate Guide to Data Labeling in Machine Learning [2024 edition]

This ultimate guide covers all the important aspects of data labeling. Find out what data labeling is all about, and how it can improve your enterprise

Data labeling is the task of systematically recognizing and identifying specific objects within raw digital data, such as video stills or computerized images (in the context of computer vision), thereby “tagging” them with digital labels that enable machine learning (ML) models to create accurate forecasts and assessments. In the context of natural language processing (NLP), data labeling would refer to text annotation: classifying words/phrases, identifying relations, and extracting information.

Data labeling and data annotation techniques are characteristic of the initialization stage when developing a machine learning model. It requires the identification of raw data, and then adding one or more labels to that data, to specify its context for modeling. This preprocessing stage permits the machine learning model to assemble accurate predictions.

For example, digital data annotation can permit autonomous automobiles to stop when approaching pedestrian crossings, digital assistants can recognize certain human voices, and digital security cameras can notice and witness strange or suspicious conduct.

Machine Learning has made significant advances within the last twenty years or so. This has been attributed to concurrent and contemporary improvements in computing processing power and the major breakthroughs achieved in deep learning research. Another key catalyst has been the quantity of digital data that has been collected and stockpiled. These advanced algorithms have progressed rapidly, and their need for digital training data has kept pace, too.

In the past two years, the data labeling process has seen progress with the popularity of foundation models and generative AI being able to work on specific data labeling tasks. Today, you can use models such as GPT or OWLV2 to do zero-shot labeling so data labeling teams can focus more on quality assurance. More on that later in this guide.

A key concept required for successful machine learning outcomes is the technique called supervised learning. Supervised learning is a machine learning approach defined by utilizing labeled datasets. These datasets are designed, over time, to train or “supervise” algorithms to correctly classify data, thereby predicting accurate outcomes. The machine learning model can measure its accuracy by processing the datasets and, therefore, self-learn over time.

How Precisely Does Data Labeling Work?

Data Collection Process

Collecting significant digital data – images, video stills, videos, audio files, transcribed texts, documents etc. – is typically the first step in the process. A large and mixed quantity of digital data will guarantee more accurate computerized results in the long run than a diminutive amount of digital data can provide. Your sources could come from public datasets, proprietary data, or synthetic data generation.

Data Labeling

Labeling data was traditionally a process where human labelers were employed to identify entities in unlabeled data. This is done using a data labeling training platform. For instance, people might be asked to label digital images to determine if there are any humans in them or to track a tennis ball during a tennis match in a video.

Today, the data labeling process has been sped up with the help of machine learning models, foundation models, and generative AI. These models can take input data and assign labels following a presrcibed ontology through either does zero-shot labeling or assisted labeling.

For example, in image segmentation for computer vision use cases, assisted labeling is where humans click on a specific image in a digital asset, and then a model embedded in a data labeling platform predicts the boundaries or segments of objects within the image based on the clicked points.

Zero-shot labeling through GPT:

Assisted labeling through SAM:

Quality Assurance Methods

Quality assurance in data labeling is crucial to ensure the accuracy, consistency, and reliability of labeled datasets, which are essential for training high-performing machine learning models. Several strategies and techniques are commonly employed to conduct quality assurance in data labeling processes: developing clear and comprehensive annotation guidelines, regularly evaluating annotator performance, creating gold standard data (or Honeypot), calculating inter-annotator agreement, and a variety of quality control checks. We made a guide to top data metrics used for assessing labeled data quality.

What Factors Affect the Quality of Data Labeling?

Information and Context

With data labeling, there is a key requirement for basic domain knowledge and contextual understanding within the labor workforce. These skills are mandatory to be able to construct high-quality, structured digital datasets for ML business models. Labelers with expertise in the specific area (e.g., medical images, street scenes for autonomous driving) can make more informed decisions, leading to more accurate labels. The following methods are helpful whether you are working with in-house labelers or a professional data labeling workforce.

Annotation Guidelines: Detailed and clear annotation guidelines ensure consistency among labelers, reducing variability in how different people interpret the same data. Guidelines should include examples, edge cases, and decision-making criteria.
Training for Labelers: Beyond basic domain knowledge, specific training sessions for labelers on the tools and the project's nuances can improve labeling accuracy and consistency.

You can learn more about datasets through these useful links: How to create a dataset for Machine Learning ; How to clean a dataset in python

Agile yet Inflexible

ML business modeling is an iterative process. Data labeling will naturally evolve as the ML business models' constructive testing and quality validation develop, and key learnings are adopted from their outcomes.

Managed data labeling teams must be flexible enough to merge changes that adjust to meet the end users' changing needs or products. A robust, yet flexible, data labeling team should be able to react positively to data volume changes, increases with task complexity, and task duration shifts. The more adaptive the labeling team can be, the more successful the ML business projects will be.

Two ways to facilitate for this:

Version Control: Implementing version control for datasets allows tracking changes over time, facilitating the integration of new data or corrections to existing labels without losing the history of modifications.
Scalability of Labeling Efforts: Technical infrastructure that supports scaling up the labeling process efficiently is crucial. This includes tools that automate parts of the labeling process (e.g., pre-labeling with machine learning models) and platforms that facilitate distributed labeling efforts.

Communication is Key

Positive and direct communication within the data labeling team is vital. Any practical and closed feedback loop is an immaculate means to establish dedicated communication and cooperation between the ML project team and the data labelers. In addition, data labelers are always advised to share their key learnings as they label the data, so these valuable insights can be adapted to modify the approach if required.

To boost this even further:

Quality Control Mechanisms: Automated and manual quality control mechanisms are critical. Techniques such as inter-annotator agreement metrics and spot-checking by expert reviewers help maintain high labeling standards.
Machine Learning Assistance: Utilizing ML algorithms to pre-label data or suggest labels can increase efficiency and consistency. However, careful oversight is necessary to ensure that biases or errors from these algorithms don't propagate through the dataset.

Some Common (and Important) Missteps to Avoid During The Data Labeling Process

Once a data labeling team is established, performs its duties, and commences delivering valuable results, it can easily get carried away when deep-diving into data labeling for any ML business project. Instead, one needs to be mindful of the missteps and pitfalls that may derail any ML project – and certain principles or rules must be adhered to when growing the digital data labeling enterprise.

Maintaining consistency throughout the labeling process

When conducting data labeling, it's crucial to ensure that the process of finalizing tags seamlessly aligns with the established and agreed-upon formatting standards. Any additional tags needed should be communicated to the entire team in advance.

For instance, when annotating text, issues like inconsistent whitespace and punctuation can arise, which may impact the accuracy of the annotations. Fortunately, various tools are available to address and rectify such discrepancies efficiently.

Throughout any data labeling endeavor, various challenges are bound to emerge, necessitating solutions along the data annotation journey. However, some challenges may signify underlying issues that require deeper attention.

For example, in the medical imaging industry, annotators must possess a high level of expertise in medical knowledge and practical experience. This expertise is particularly crucial when annotating intricate images derived from X-rays, MRI scans, and CT scans. Ensuring that the labeling workforce possesses the necessary skills and expertise is paramount to maintaining the quality and accuracy of the labeled data in such specialized domains.

Additionally the use of two key concepts in data labeling: consensus and programmatic error spotting can help maintain the consistency and quality of labeled data. We'll discuss these further down the article.

Complex and layered ontologies in data labeling

When grappling with the intricacies of data tagging, particularly in the context of complex and layered ontologies, further challenges may emerge beyond the scope of simple nested or inverted data tags. Consider a scenario involving a text document: initially, a tagged sentence or phrase may appear correct, as the digital tags were applied according to the sequential order in which they appeared within the sentence. However, complications arise when the exact words or phrases occur in different orders but retain the same meaning. In such cases, the arrangement of tags could vary significantly, leading to divergent interpretations of essentially identical content.

This issue underscores the importance of ensuring consistency and alignment across all resources involved in digitally labeling machine learning datasets, particularly when utilizing nested labels or operating within complex ontological frameworks. It extends beyond textual data and is equally applicable to other data modalities, such as images and videos.

Complex and layered ontologies introduce additional layers of abstraction and organization to the tagging process, enabling more nuanced categorization and representation of data. We'll discuss tackling these challenges further in the article.

Learn more!

Discover how training data can make or break your AI projects, and how to implement the Data Centric AI philosophy in your ML projects.

Download White Paper

Adding Any New Labels to Your Project Mid-Term

Halfway through your first data labeling iteration, your team may realize there will be a need for a new tag that is currently not in the agreed-upon master labeling list. In one initial undertaking, the team might simply add a new required data label to the master list, informing the whole team to use this new digital label when required.

However, this process should be highly discouraged, and the simple reason is that the labeled data the team has already marked would have to be re-examined to see if the new label suddenly fits into the already label-processed digital data. It is best practice always to compile and agree upon the list of labels before commencing any labeling endeavor. If a new data label is devised as a required mid-project, it is ideal to note this occurrence and then review – as a team – how it was missed after the data labeling exercise is completed and how this discovery can be prevented from re-occurring beforehand.

Top methods for label error detection and quality assurance in unstructured data

Here, we will cover the top methods for label error detection in unstructured data. You might be surprised to find that there is a range of methods, and they don't all have to be model-based! Most are workflow-based!

Why is error detection important?

Label error detection is part of the quality control process for model development. In structured data, there are generally more or less objective measures for model performance (e.g., precision and recall, regression measures, separateness measures), but for unstructured data, there is an additional element of human interpretation of both the model outputs and the data inputs and what is subjectively or even semantically “correct.”

Why do error detection?

To better understand the data: unstructured data is typically much harder to summarize than structured data. Serving up examples of where labels are uncertain or “bad” can help inform the MLOps Engineer where to focus on data generation efforts, though this process, as you will read, is model-based and therefore can have a higher bar to clear in implementation.
To understand if there are underlying issues in labeling instructions: it can quickly become apparent that labels aren't as you intended them to be if there are certain consistencies in the errors. This is oftentimes due to a lack of clear instructions, especially if there is ambiguity in how to label a certain class.
To understand if there is labeling bias: unconscious bias can creep into datasets, especially when dealing with qualitative, emotive, and cultural label classes. While labels may be accurate for the subjects involved, they may not be representative of all subjects belonging to that class (e.g., querying “successful business person” returns only images of middle-aged white men dressed in suits could be construed as a severe bias issue as the model/labels are omitting—or have no examples of—data where successful business people of other ages, races, genders, and fashion-senses exist)

Prevent errors from the start - interface setup

Preventing errors can be fairly simple, especially when using a labeler-centric platform like Kili Technology. Within the Kili Technology platform, you have the ability to structure labeling jobs that can force compliance with the desired outcome. Tools such as required jobs, specified relations, and handy pop-up tooltips to quickly clarify annotation intent to the labeler are all standard within Kili.

object-detection-job-is-required-specifying-level-of-occlusion-is-not

Object Detection Job is Required, specifying level of occlusion is not

object-relation-between-specific-objects

Object Relation can either be between specific objects or relate any object to one another

Tools like these can help prevent simple errors, smooth out the workflow, and help guide labelers to the correct output.

Catching errors early - quality control workflows

If you’re curating your own dataset from scratch or relabeling a standard dataset, then you have the opportunity to both define what the ground truth is and catch errors at the earliest stage. This is also the best and most cost-effective way of detecting errors—especially for novel/purpose-built AI/ML applications.

A quality control workflow can be as simple as reviewing a randomly chosen subset (say, 100 assets) of the labeled data to validate that the labels are what you expect them to be. If there is higher than a given threshold of errors (say, 10) in your subset, it may warrant further investigation.

quality-control-workflows-in-kili-technology-platform

A more complex review procedure, keep reading to learn more!

Use analytics to spot inconsistencies in human labels or model predictions

While it is technically more total work, the benefit you get from labeling the same asset more than once is a greater degree of certainty that it was labeled correctly, and the opportunity to quickly sift through data based on the level of consensus which can greatly focus the review process covered above.

Consensus works by having more than one labeler annotate the same asset. When the asset is labeled, a Consensus score is calculated to measure the agreement level between the different annotations for a given asset. There are various methods for calculating consensus for different data types. In Kili Technology you can also specify custom consensus calculations to suit your specific needs.

Once you have a calculated consensus score, it is easy to sort by and query your dataset to empower a focused review of data below certain thresholds for agreement. In this way, you are leveraging a bit more up-front work, for more ease of dataset use further down the pipeline.

Much like a human-human consensus, you can also evaluate your model based on data labeled by humans (test and/or validation subset). The calculations for this agreement metric are the same over the data types, and in the Kili Technology Platform, you can easily bubble up the least-agreeing assets for further investigation.

This can help you identify where the model is struggling, and find commonalities in the disagreement between the machine and the annotator. This type of metric can also be used in a reverse way—having a validated “good” model spit out inferences and test new annotators against the machine, especially useful when using the same testing dataset that is known to be labeled well by your model.

Using gold standards

example-of-a-gold-standard Integrating gold-standard datasets offers an additional layer of quality assurance and validation. Gold standard datasets serve as benchmarks for evaluating the performance of label error detection methods. By comparing the results of error detection against a known gold standard, organizations can assess the accuracy and efficacy of their labeling workflows.

Incorporating gold standard datasets provides a ground truth or reference point for verifying the correctness of labeled data. It enables organizations to identify labeling discrepancies and errors more effectively, enhancing the overall quality assurance process.

Gold standard datasets facilitate iterative improvement by serving as a basis for refining labeling guidelines, training labelers, and fine-tuning error detection algorithms. Organizations can leverage insights from comparing labeled data against the gold standard to iteratively enhance the accuracy and consistency of labeling practices.

Program a custom plugin to check certain rules

If you want to catch some simple errors quickly and automatically, it is easy to program a custom plugin in Kili to perform checks (regex, number of objects, name, and signature, etc.) upon certain triggers (i.e., submitting an asset, reviewing an asset). You can also compute custom consensus if you’re not finding the option you need in the Kili Technology's standard!

program-custom-plugin-in-kili-technology-platform

Automating the review of tedious rules in the Kili Technology training data platform

This is a powerful way to automate the review of tedious rules that could trip up even the most seasoned reviewer.

Further, plugins like these can automate workflows as well, as the triggering event can cascade to any number of next state(s) for the asset—they can be returned to the annotation queue, flagged for review (1st, 2nd, nth-order reviews are possible!), provide automatic feedback to annotators, or even sent to another labeler for an additional set of labels for more robust consensus (including best 2-of-3 consensus, etc.)

two-stages-review-process-for-DCAI-model-in-production

The two-stage review process for a DCAI model in production]

Using model confidence to help you spot errors

If you’re reading this, you’re likely an ML Engineer or Data Scientist—or you know one. So I’ve saved the most technical for last. You can of course take the approach of having your model give you its “confidence” in its prediction. While it is tempting to eliminate as much human labor as possible through the power of coding and cheap, computing, it is best to try to implement the first four methods before diving into the code (you’ll catch things with less effort, and way less frustration with your model).

Ok, now that we have that disclaimer out of the way, let’s talk about model confidence and the many ways that it can be implemented.

To implement machine learning model confidence in a data-centric AI workflow, there are a few key steps to follow:

First, you will need to collect and prepare the data that will be used to train the machine-learning model. This may involve cleaning and preprocessing the data, as well as annotating it to make it intelligible to machine learning models.
Next, you will need to select and train a machine-learning model that is capable of calculating model confidence metrics. Some models use a probabilistic approach, where the confidence is calculated based on the probability of a given prediction being correct. Other models use more complex methods, such as comparing the output of multiple models or using uncertainty estimates derived from the training data.
Once the model is trained, you must evaluate its performance and determine its model confidence metrics. This may involve using tools and algorithms from the Cleanlab github repository, or implementing your own methods for calculating model confidence.
After the model confidence metrics have been calculated, you can use them to improve the model's performance. For example, you may choose to use the confidence metrics to weight the predictions in your DCAI workflow and perform active learning by focusing on the least-confident predictions first, correcting them to enrich the dataset, and finally re-training the model and iterating.

By following these steps, you can implement machine learning model confidence in a DCAI workflow, improving the performance and reliability of your AI systems. At Kili, we have several notebooks that implement Auto ML seamlessly integrated into the Kili workflow, and some even have model error detection modules—check them out for yourself!

automl-error-detection-modules

AutoML in the Kili Technology Platform

In conclusion, label error detection methods are crucial for ensuring the quality and accuracy of labeling workflows in machine learning. These methods involve setting up interfaces that allow for easy and efficient labeling, implementing quality control workflows to identify and correct errors, using analytics such as consensus to determine the reliability of labels, using custom plugins to check for adherence to specific rules, and leveraging model confidence metrics to assess the accuracy of the labeling process. By utilizing these label error detection methods, organizations can improve the reliability and effectiveness of their machine-learning models.

Data labeling best practices

Acquiring high-quality digitally labeled data is an expansion barrier that becomes more significant when contemplating the handling of complex ML models. Yet, there are many strategies and approaches to improve the efficacy and precision of the data labeling process:

Clear Labeling Instructions: Communication with the digital data labelers and providing clear labeling requirements will ensure the desired accuracy of the delivered results.
Label Verification: It is necessary to audit the digital labels to confirm their exactness and make any adjustments if required.
Learning Transfer: Another method to improve the efficiency of data labeling is to reuse previous data labeling assignments to create hierarchical data labels. Essentially, ML operators can utilize the output of previously digitally-trained ML project models as input for another new project.

Data labeling and foundation models

From data to tasks with foundation model

The advent of foundation models has introduced significant advancements in data labeling, offering ways to streamline and enhance the process. Combining these powerful models' strengths with human expertise makes achieving more accurate, consistent, and efficient data labeling possible, paving the way for more advanced and capable machine-learning models.

Semi-Automated Labeling

As mentioned earlier, foundation models can be used to pre-label data, significantly speeding up data labeling. For instance, a model pre-trained on a large corpus of text can be fine-tuned to identify specific entities in text data, which can then be reviewed and corrected by human labelers. This semi-automated approach reduces the time and effort required for manual labeling while maintaining high accuracy levels through human oversight.

At Kili Technology, you can use the labeling copilot to do zero-shot labeling or bring-you-own-model to use your custom model to further label and refine its performance.

Less data to label

Traditionally, supervised learning has required large amounts of labeled data. Foundation models, however, can be fine-tuned with relatively small amounts of labeled data, thanks to their extensive pre-training. The generalizability of foundation models allows them to be adapted across different domains with minimal fine-tuning. Once a foundation model is fine-tuned for a labeling task in one domain, it can be relatively easily adapted to similar tasks in other domains, leveraging the knowledge it has already acquired. Check out our handy guide to fine-tuning foundation models for more info.

Conclusion

AI is revolutionizing the course on how we perform certain repetitive functions, and enterprises that have adopted such business models are gaining benefits. The technological opportunities AI can deliver are potentially inexhaustible and will assist diverse industries in becoming smarter: from medicine to agriculture, to recreation and sports. Data digital annotation is the first stage of such innovation.