Data Labeling and Large Language Models Training: A Deep Dive

For those not well-versed in machine learning, large language models like GPT-3.5, which ChatGPT is based on, seem self-sufficient. These models are trained with unsupervised or self-supervised learning. Simply put, it requires minimal manual intervention to produce a model capable of conversing like humans.

This begs the question – is data labeling still relevant for large language models? 

It’s unwise for machine learning teams, project managers, and organizations to dismiss the importance of data labeling. On the surface, large language models (LLMs) seem capable of taking on any task, but reality paints a different picture.

In this article, we’ll explore the mutual-beneficial relationship between data labeling and large language models. 

What is Data Labeling 

Data labeling, or data annotation, is a process of identifying, describing, and classifying specific elements in data to train machine learning models. The labeled data then serve as a ground truth for the foundational model to process, predict and respond to real-life data. It helps the model, or neural network, learn and make decisions that produce the desired results. 

For example, a document imaging system needs to identify personal identifiable information in the raw data. To do that, labelers annotate the name, ID, and contact details on the training sample. Then, machine learning engineers train the model with the dataset to enable entity recognition and extract personal details from stored documents. 

Data labeling seems straightforward, but various parameters might affect the annotation outcome and the model’s performance. Hence, ML teams use data labeling software to support their effort in creating accurate and high-performing models. 

Data labeling use cases

Data labeling has been pivotal in training machine learning models long before the emergence of LLMs or generative AI. For example, 

• ML engineers label data to support natural language processing (NLP) tasks such as named entity recognition, translation, and sentiment analysis.

• Annotation is also helpful in training image recognition systems to detect and classify objects.

• Healthcare systems train neural networks with annotated data to diagnose diseases from imaging data.

• The finance industry trains models with diverse datasets to perform fraud detection and credit scoring.

• Self-driving cars depend on accurate datasets to train models capable of analyzing multiple sensor data in real-time. 

What are Large Language Models? 

Large language models (LLMs) are linguistic neural network models trained with massive amounts of data. They are mostly based on the transformer neural network architecture. Unlike its predecessor, the transformer is capable of focusing on multiple words in parallel, which allows the model to understand how words that are far apart relate to each other. 

Linguistic models like GPT, Bard, and BERT are large language models. As such, they can converse with users and construct answers in natural languages. However, their intelligence comes at a steep cost, and the sheer amount of raw data required. Despite that, there are limits to what those models can do in downstream applications, which we’ll explain below. 

How are Large Language Models Trained? 

Before we move further, let’s look at how machine learning engineers train LLMs like GPT-3.

Preparation and Preprocessing

First, data scientists curate a large amount of raw textual data from various sources, including the internet, books, and public datasets. Then, annotators clean the data to ensure they are free from errors, noise, or bias. This follows the conversion of raw data into formats the model understands with preprocessing steps like tokenization. Tokenization turns text data into smaller linguistic units consisting of words or subwords. 

Training and Optimization

Large language models like GPT3 use next-token prediction and mask-level modeling to develop a natural understanding of linguistic structures. 

• Next-token prediction enables the model to predict the most likely word or token that comes after the currentone. For example, the model attempts to complete the phrase ‘The ocean is ___” with the word ‘blue’. 

• Mask-level modeling involves removing a specific word or phrase in a sentence randomly. Then, the model is prompted to predict the appropriate word or phrase that fits into the blank. 

Both natural language processing training methods enable the model to improve its probability of providing the most relevant output. It does so by comparing the generated output with the desired outcome. Then, it passes the differences or errors to the network for subsequent optimization. The model can readjust its parameters and weights by evaluating its loss gradient to bridge its generated differences. This process is iterative, and the model repeats the steps until it achieves satisfactory performance. 

Model evaluation and fine-tuning.

Finally, the trained model undergoes evaluation testing, where ML engineers feed it with a set of annotated test data. Depending on the result, they might further adjust the model’s parameter or proceed to finetune it for specific purposes. The latter involves supervised training, where the model is fed with annotated datasets. 

The Significance of Data Labeling in Training Large Language Models 

Theoretically, large language models can assume functionality without labeled data. Models like GPT3 use self-supervised or unsupervised learning to develop capabilities of understanding natural languages. Instead of training with annotated data, it uses the next-word prediction approach to learn how to complete sentences in the most contextually-logical manner.

As intelligent as they are, large language models are imperfect and often unsuitable for practical applications. Here’s why.

• By itself, a large language model cannot perform specialized or business-specific tasks. For example, you can’t use ChatGPT as your company’s chatbot as it was not trained on your products or services.

• LLMs are prone to bias, which affects their response's accuracy and appropriateness. This may result in costly unfair predictions in security, finance, and other mission-critical industries. 

• Models like GPT can be abused or misused without additional regulations. As pre-trained models, they might respond with inappropriate text, outdated information, or made-up facts.

Generally, LLMs are dependent on the datasets they’re trained on and their ability to self-supervise. Unfortunately, there are gaps between the desired output and actual performance in real-life models, which brings data labeling into play. Rather than training the entire model, human labelers help optimize the model for practical applications. 

Below, we share how annotation benefits the LLM’s overall performance, correctness, and practicality. 

Pre-training 

While large language models don’t use annotated data directly during training, it still benefits from human labelers. Often, such models are too costly to retrain, underscoring human annotators' importance in limiting errors. Annotators curate and clean the dataset from noise and errors, which improves the trained model’s reliability.

Fine-tuning

Data annotation is critical to tailoring large-language models for specific applications. For example, you can fine-tune the GPT model with in-depth knowledge of your business or industry. This way, you can create a ChatGPT-like chatbot to engage your customers with updated product knowledge. 

Model evaluation

With researchers continuing to introduce newer linguistic models, they need a fair way to evaluate their performance. Annotated data provides a single ground truth to compare metrics like precision, recall, or F1 score between models. 

Context understanding

Large language models are generally superior in understanding different linguistic contexts and nuances than their predecessors. With that said, not all models are equally adept at understanding the intricacies of human languages. Therefore, annotation helps to enhance their capabilities in understanding and responding to different language styles. 

Build high-quality datasets to train your LLM

LLM performance depends on the quality of training datasets. Build the best dataset for your ML project with our platform today.

Book a demo

Three steps to fine-tune LLMs using data labeling: ChatGPT’s example 

A pre-trained GPT model is capable of stringing sentences together coherently but requires further refinement to fit specific purposes. ChatGPT was finetuned using a technique called reinforcement learning from human feedback (RHLF) to improve its purpose alignment. RHLF works on the principle of rewarding the pre-trained model as it predicts probable output from labeled datasets. 

To fine-tune ChatGPT, OpenAI’s engineers went through these steps.

Step 1: Supervised Fine Tuning (SFT)

SFT involves a team of human annotators creating a set of prompts and their expected outputs. Then, the engineers train the foundational GPT model with the prompt-outputs combination to produce an SFT model. As this method is done manually, it is cost-intensive and lacks scalability. Also, the dataset created by labelers is insufficient to thoroughly fine-tune a model as large as GPT, which brings us to the next step.

Step 2: Reward Model

The reward model overcomes the SFT’s ability to scale. Instead of creating datasets from scratch, engineers use the SFT model to generate several answers automatically for a prompt. Then, the annotator ranks the answers to reflect the best to the least desirable match.

With this method, OpenAI’s engineers can generate a much larger dataset without being limited to people resources. They then use this dataset to train the reward model to predict the scores of the SFT model based on human preferences.

Step 3: Proximal Policy Optimization (PPO)

In the final stage, the engineer creates a reinforcement learning mechanism that involves the PPO and the reward model. The PPO model, a copy of the LLM model, generates an output to which the reward model assigns a score. According to the score, the PPO model would adjust its policy and improve its performance in the next iteration. 

Automating Data Labeling with LLMs 

ChatGPT’s use of data labeling highlights the latter’s importance in aligning LLMs for practical purposes. On the same note, it also shows that LLMs prove useful to scale labeling tasks. Companies are under cost and resource pressure to create large and accurate datasets for increasingly-complex deep learning models. By enabling ML-assisted labeling, they can be more productive and cost-effective in annotation tasks. 

How machine learning can help automate data labeling

LLMs are capable of comprehending human languages naturally. However, they are equally adept at structuring data, active learning, and generating synthetic data. This makes LLMs like GPT helpful in pre-annotation tasks. Instead of labeling the entire dataset manually, annotators can use LLMs to identify and label specific entities in textual data. 

Some LLMs are capable of zero-shot or few-shot learning, which enables the model to make predictions without any need for finetuning. In data labeling, this capability allows LLMs to perform manual labeling tasks without lengthy preparation. This way, human labelers can devote more time reviewing the annotated dataset. Naturally, this translates into time and cost savings for organizations. 

Tools and techniques for automating data labeling

You can now leverage ChatGPT or GPT itself to assist you in pre-annotation tasks. To do this, you need prompt engineering knowledge to create the appropriate instruction for ChatGPT to analyze, annotate and return the results in the appropriate format. Alternatively, ML engineers can programmatically access the GPT model with an OpenAI API key.

Let’s take the example of a NER project. In such a context, the LLM can be asked to return the entities as a JSON string. Once under this format, the entities can then be uploaded to labeling platforms such as Kili Technology. 

This enables a gapless integration of LLM with an existing automated labeling pipeline. For example, we demonstrate how to apply named entity recognition tags on the CoNLL2003 dataset with GPT and export the pre-annotated data to Kili Technology  in this guide.

Using Kili Technology to label data with LLMs and Foundation Models

Kili Technology now natively supports OpenAI's GPT to pre-annotate textual data so data labeling teams and data scientists can focus on improving the quality of their datasets. Now, teams can select batches of text and get labeled data in a few clicks.

Additionally, Kili's machine learning experts have developed handy recipes and tutorials to use with Kili's Python SDK so more advanced users can put LLMs and Foundation Models to work.

Simplify your LabelingOps

Integrate labeling operations on Kili technology with your existing ML stack, datasets and LLMs. Let us show you how.

Book a demo

Wrapping Up 

The emergence of large language models has shifted data labeling’s traditional role in supervised machine learning. While LLMs can train on massive unlabeled datasets, human labelers remain essential in developing next-generational AI applications like ChatGPT. Likewise, data labeling process has gotten a boost from LLM’s data generation and prediction capabilities.

Talk to our team to learn more about reducing data labeling time with large language models.

References

https://www.assemblyai.com/blog/how-chatgpt-actually-works/

https://openai.com/blog/chatgpt

https://huggingface.co/learn/nlp-course/chapter1/4

https://lambdalabs.com/blog/demystifying-gpt-3

https://www.forbes.com/sites/bernardmarr/2022/09/30/the-problem-with-biased-ais-and-how-to-make-ai-better/?sh=623bf0c54770

https://smartdesignworldwide.com/ideas/train-gpt-on-proprietary-data-for-business/

https://huggingface.co/blog/rlhf

https://bdtechtalks.com/2023/01/16/what-is-rlhf/

FAQ on data labeling and LLMs