Automatic error detection for image classification in Kili

Few-shot object detection: a fine-tuning approach

Get Started

With the increase in the complexity of AI-based projects and the huge increase in the sizes of the datasets used, it became painful and sometimes unfeasible to use the traditional pipeline of labeling and training. Also, the increase in the number of hyper-parameters of the models makes it pretty hard to manage the training experiments and to compare the performance among various trained models. In this article, we will explain a very powerful couple of tools to produce a comprehensive  AI training pipeline. The first one is Kili 

. To be more specific, the article will take into consideration the surface defect detection task using the scratches class of the 

First of all, let’s start by understanding the concept of data-centric AI, which is an artificial intelligence approach that focuses on the data to make sure that it represents what the desired AI system should learn. Also, the concept of data-centric AI prioritizes the quality of the data over its quantity. In other words, instead of enhancing the structure of the model that will be trained, this approach suggests enhancing the quality of the data to improve the performance of AI-based systems.

AutoML is a tool provided by Kili that gives the ability to train models and to use them to label the rest of the data automatically. In this way, we save labeling time and make the labeling process faster and more consistent. For now, AutoML supports natural language processing tasks such as Named Entity Recognition and Text Classification and Computer Vision tasks like object detection and image classification.

W&B is a product that provides a dashboard to monitor the whole training process. It embeds various visualizations to compare the trained models and trace the performance of the models. Using W&B, we can understand the effect of the parameters used for training among all training experiments.

Surface Defect Detection is the task of detecting defects on surface pictures and classifying the detected defect into specific classes.

NEU-DET is a dataset provided by Northeastern University. It contains 6 classes of defects with an average of 240 images for training and 60 images for validation per class.

To achieve this task, we can use the traditional pipeline which suggests we start by constructing the dataset, after that, completely manually labeling it one by one, and finally, training the desired model.

If we investigate the following figure, we notice that it is suitable only for simple tasks, but for complicated tasks, it is not feasible to annotate the data completely in a manual way. 

To solve the problem explained in the previous paragraph, Kili provides AutoML to automate some parts of the process of labeling which leads to speeding up the whole process. 

By using AutoML we start the training process by collecting the unlabeled data; then, we import the dataset into Kili; then we label only a part of the dataset in the labeling interface. After that, using the `train` command, we obtain the first model. If we get partially satisfied by the model, we call the `predict` command to automatically label another part of the dataset so the labelers do not need to label that part but only need to verify it. 

Labeling a photo that contains 3 different objects needs the following steps:

Therefore, it needs a minimum of 13 seconds. Conversely, verifying the labels of an image that contains 3 objects annotated with a partially good model needs the following steps:

As a result, we notice that by importing the auto-generated labels, the time of drawing and choosing the class will be reduced, and by the improvement of the model over time it will be negligible. Therefore, using AutoML can potentially reduce the time of labeling by a percentage of about 80%.

To conclude, the following figure describes the AutoML-empowered pipeline. 

Even though the improvement suggested by Kili is speeding up the process of training, the previous pipeline still has an issue. In complex tasks of supervised learning, we need to fine-tune the hyper parameters of the models. Therefore, this leads to the need for an effective way to report and compare the performance of trained models to choose the best one.

Weights and Biases provides a solution that can help us report and compare models during the training process. W&B provides a dashboard that is associated with a project to visualize the changes in the performance due to various parameters among different models. 

The good news here is that Kili-AutoML supports W&B, so the `train` command is integrated with W&B. Therefore, we can use the power of W&B while we are training our models using AutoML. The following figure describes the final pipeline that is empowered by AutoML and W&B

Firstly, we can start by creating a new project on Kili platform like the following:

Later on, we upload our unlabeled dataset and then we specify the classes that we have.

Finally, we end up with a project dashboard that contains the uploaded assets with the ability to filter and sort them according to last update, labeller and status.  

Subsequently, we start to annotate our dataset by labeling a small part of the assets that we imported to the workspace. Kili provides a smooth interface for labeling with very practical keyboard shortcuts. Also, it allows us to inspect the images with different brightness and contrast values and to rotate the images.

Basically, to be able to run the `train` command we need to acquire a Kili API key and the ID of the Kili project we are working on; this can be acquired from the Kili application settings, and can be passed directly to the Kili commands or set as an environment variable. After manually labeling a part of the data, we can call the script in the following way after cloning the project and installing the required packages:

kiliautoml train --api-key YOUR_API_KEY --project-id YOUR_PROJECT_ID

 detection model and by the end of execution of the command, we get reports about the performance in both the terminal and W&B dashboard:

If we check the logs on the W&B dashboard, we can see that it provides several types of evaluations, both per iteration and global. For example, the following image shows the evaluation metrics for one iteration:

Also, we can see the visualized loss of training and validation in the loss section:

For more insightful reports, W&B embeds a section for inference samples that shows the output of the model over some samples from the validation set like the image below:

W&B also provides sections like training batch mosaics, confusion matrix, F1 curve, PR curve, P curve, R curve, learning rate graph, and bounding box debugger. In addition, to be able to monitor the training experiments, W&B provides detailed reports and comparisons between the intermediate models. For example, the following chart reports the evaluation metrics of all trained models; each curve is associated with the ID of the experiment:

Similarly, W&B provides comparisons between the losses of different training experiments as a chart like the following:

Finally, W&B gives reports about the training environment and the used hardware usage. For example, the following charts report the usage of the GPU of the training machine:

W&B embeds further insights about the system like GPU memory allocated, GPU time spent accessing memory, GPU temp, GPU utilization, network traffic, disk utilization, CPU threads in use, memory available and CPU utilization. For better portability, at the end of our work, we can use the “Create report” button to export all the charts as a report that can be downloaded in LaTex or PDF formats.

If we get partially satisfied by the model, we can call the `predict` command in the following way:

kiliautoml predict --api-key YOUR_API_KEY --project-id YOUR_PROJECT_ID

Make Training Process Productive With Kili AutoML and Weights & Biases

Recent advances in supervised learning owe, partly, to the availability of large annotated datasets. For instance, the performance of modern image classifiers saturates only with millions of labeled examples. Assembling such datasets typically requires the extensive labor of human annotators. Platforms like the one offered by Kili Technology allow efficient annotation of data points (or assets) by multiple people at the same time. We wish to output a dataset with a single label for each asset. However, for efficiency and robustness,  the annotation process is often done simultaneously by several labelers. Then, several labelers might label the same asset differently, introducing a noisy dataset. This noise might come for various reasons, such as unclear instructions, naturally ambiguous labels, skill differences among labelers, or just human mistakes. 

Commonly, some super-annotator (or reviewer) will be tasked to check if the data has been labeled correctly and to select the best label to use in the final dataset. However, this method has its limitations, as it adds another layer of costly manual labor when creating a labeled dataset. This super-annotator also commonly only sends feedback to the annotators, creating a correction loop that takes time and slows down the annotation process.

This problem has drawn some attention in the scientific literature, as it relates to the problem of crowdsourcing knowledge: how to retrieve truth from many noisy and unreliable estimators. For the problem of unifying annotations from concurrent labelers, the scientific literature extensively studied the case of binary and multiclass classification. The problem is tackled through several algorithms, having their own advantages and drawbacks. They can be simple (computationally cheap) –like the majority vote 

, or more complex but yielding better results –like the Dawid & Skene estimator 

Classification is a supervised learning problem, where the goal is to predict the label of a data point among a pool of predefined labels.

Let us remind ourselves that when building a training dataset, we do not have access to the ground truth of labels. We only have access to the labels given by labelers, which might contradict each other. Not all labelers annotate all data: the amount of annotations given to each label is heterogeneous. This corresponds to the reality of crowdsourced annotators, who will not necessarily annotate the whole dataset, or do not have the knowledge to annotate some elements of the dataset. In other words, different labelers will label every data point (or asset) at least once.

In this review, we will study methods that output a unified label per data point from several labels for the case of a classification task. First, we will describe more precisely the unification problem. Then, we will examine three different algorithms that solve the task: the majority vote, Dawid & Skene estimator, and finally, the GLAD algorithm.

In the case of a classification task, the datasets are often aggregated by applying a simple majority vote. Academics have proposed many efficient algorithms for estimating the ground truth from noisy annotations. Research addressing the crowd-sourcing problem goes back to the early 1970s’ to compile patient records. Dawid & Skene 

 proposed a probabilistic model to jointly estimate worker skills and ground truth labels and used expectation maximization (EM) to estimate the parameters. Whitehill et al. 

, proposed a generalization of the Dawid-Skene model, e.g., by estimating the difficulty of each example.

We can try to better understand the problem with an example. Let’s suppose we attempt to classify images in a binary classification problem with categories 0 and 1. Let’s further suppose that we have 5 labelers working on the project simultaneously. The output of their work might look like the following table:

For this to be useful to train a model, we want to output a single answer (0 or 1) for each image. In this setup, we could output a dataset that looks like this:

This output dataset was not done randomly, but by following an intuition: for each image, we would like to choose the label that makes the biggest consensus. Since all labelers seem to agree to classify image 1 as a ‘1’, it would make sense to output a ‘1’ in the final dataset. In the case of binary classification, we can sum up that intuition by this formula:

 the function that associates to an asset the label decided by labeler

is the easiest way to aggregate labels in the case of a classification task. The idea behind the majority vote is that for a given asset, the final label chosen will be the label chosen by the largest number of annotators. 

This is depicted in figure 1. In this figure, we want to annotate images 1,..,n, without having access to the ground-truth labels Z1,...,Zn. Each labeler gives a label for an image, Lij means labeler i annotated image j. Then, we combine the given labels to output the predicted labels Z1,...,Zn. 

This very simple method has several drawbacks limiting its real-world application. Indeed, we expect such techniques to estimate the reliability of ad-hoc anonymous labelers, the difficulty of the different items in the dataset, and the probability of the true labels given the currently available data. Indeed, some labelers might misunderstand the labeling guidelines and consistently mislabel the data. For this reason, these labelers should not weigh as much as others. More complex heuristics have been proposed to solve these issues.

Another classical method for estimating the ground-truth labels from the noisy labels is described by Dawid and Skene 

. In its simplified form, each worker is assumed to have a number ɑi to characterize 

the intrinsic labeling ability of labeler i

. For any given probability ɑi, the label from the i-th worker is correct, and with a probability of 1-ɑi , the label from the i-th worker is wrong. Workers’ abilities ɑi can then be estimated by maximizing the marginal likelihood, and an estimate of the true labels follows by plugging the estimate of workers’ abilities into Bayes’s rule.

Moreover, by regarding the true labels as latent variables, this two-stage estimation can be iteratively solved through the Expectation-Maximization (EM) algorithm 

. Various variants emerged from this algorithm, by incorporating  Bayesian inference into the Dawid-Skene estimator by assuming a prior over confusion matrices 

. This enables, for instance, to incorporate the sensitivity and specificity of the different labelers in the model.

Some more advanced estimators, like the GLAD (Generative model of Labels, Abilities, and Difficulties) estimator proposed by Whitehill et al. 

, in addition to labelers accuracies ɑi of the Dawid & Skene estimator. This setup fixes the problem where all labelers do not have the same experience level, or not the same quality of output, in case one of the labelers is an automatic classifier (or any other source of pre-annotation). The utility of modeling the image difficulty is easy to understand: if there is an ambiguous data point in the dataset (like a blurry image), then giving a ‘good’ label will be harder, or labelers might agree on a wrong label in that case, not because of lack of knowledge, but purely because of the difficulty of that particular piece of data.

Like before, the maximum likelihood estimate of the parameters of interest is obtained by  using Expectation-Maximization approach (EM).

GLAD keeps the advantages of the majority vote (small computational cost) but can recover the true data labels more accurately than the Majority Vote heuristic. It is highly robust to both noisy and adversarial labelers. Experiments show it yields more accuracy than the Dawid & Skene estimator.

To explore the importance of estimating image difficulty, a simple simulation is done in 

: Image labels (0 or 1) are assigned randomly (with equal probability) to 1000 images. Half of the images were “hard”, and half were “easy.” Fifty simulated labelers labeled all 1000 images. The proportion of “good” to “bad” labelers is 25:1. The probability of correctness for each image difficulty and labeler quality combination is given in the table below.

In Table 1, we can notice that the GLAD algorithm achieves a lower error rate than the Dawid & Skene and the Majority vote on the experiment described above, showing the importance of taking image difficulty into account.

There are various ways to solve conflicts when labeling data using multiple annotators. We reviewed 3 different ways of choosing labels in the case of a classification task, of increasing complexity. These methods all achieve the same purpose: output a single annotation for each data point when given multiple (possibly concurrent) annotations from several labelers. These methods have varying levels of complexity and accuracy but yield useful results in the case of unifying annotations for a classification task. For more complex ML tasks (Named Entity Recognition, Object detection), no unification technique has been proposed in the academic literature yet.

Alexander Philip Dawid and Allan M Skene. “Maximum likelihood estimation of observer error rates using the EM algorithm”. In: Journal of the Royal Statistical Society: Series C (Applied Statistics) 28.1 (1979), pp. 20–28.URL: 

http://crowdsourcing-class.org/readings/downloads/ml/EM.pdf 

Jacob Whitehill et al. “Whose vote should count more: Optimal integration of labels from labelers of unknown expertise”. In: Advances in neural information processing systems 22 (2009). URL:

https://proceedings.neurips.cc/paper/2009/file/f899139df5e1059396431415e770c6dd-Paper.pdf

Peter Welinder et al. “The multidimensional wisdom of crowds”. In: Advances in neural information processing systems 23 (2010).URL: 

https://proceedings.neurips.cc/paper/2010/file/0f9cafd014db7a619ddb4276af0d692c-Paper.pdf

Dempster, Arthur P., Nan M. Laird, and Donald B. Rubin. "Maximum likelihood from incomplete data via the EM algorithm." Journal of the Royal Statistical Society: Series B (Methodological) 39.1 (1977): 1-22. URL: 

https://www.eng.auburn.edu/~roppeth/courses/00sum13/7970%202013A%20ADvMobRob%20sp13/literature/paper%20W%20refs/dempster%20EM%201977.pdf

Raykar, Vikas C., et al. "Learning from crowds." Journal of machine learning research 11.4 (2010). URL: 

https://www.jmlr.org/papers/volume11/raykar10a/raykar10a.pdf

Getting a clean dataset from multiple labelers

Most performant named entity recognition (NER) models require lots of labeled entity examples to be trained on and perform well. However, annotation is a painful process that involves a lot of human effort and often demands domain expertise that is hard to obtain. To reduce this pain, pre-annotation can be used to accelerate the annotation process. Pre-annotation consists in using rule-based or Machine Learning models to automatically label datasets that are then reviewed by human annotators. In this way, by simply correcting and verifying the pre-annotations, labelers' work can be much simpler and faster. 

Pre-annotation NER models still need task-specific labeled data to work but these models are needed as soon as possible in the annotation project when manually annotated data is scarce. When labeled data is rare, traditional fine-tuning methods do not work, and more ingenious methods are needed to squeeze as much information as possible from the few annotated examples. In recent years, few-shot learning methods have been developed. While they are clearly not as good as traditional methods trained on many examples, they can be helpful in pre-annotation since they are a good tradeoff between accuracy and the number of labeled samples.

In this post, we will, first, see why traditional methods do not work in the few-shot setting. Then, we will explore several few-shot learning methods starting with prototypical methods. Then, we will see how label semantics information can be introduced in the model and end with meta-learning.

For more information about Named Entity Recognition, you can check our 

Transfer learning consists in adapting a pre-trained model that has been trained on lots of data to a specific new task. This is very common in NLP. Large language models like BERT or GPT are trained on simple tasks like 

 that allow the model to get a good understanding of language, with little annotation effort. When adapting to a specific task, the pre-trained model is reused. The last layers of the model are changed so that the output corresponds to the specific task, while the backbone is kept and initialized with the weights of the pretrained model. Then, this model is trained on the available training data of the specific task.

Nevertheless, the fine-tuning of the model still requires hundreds or thousands of examples per class to perform well. Simple training of the model on a few labeled examples is not enough to get a good generalization. As shown in figure 1, the model overfits if it only has 1-5 examples per class.

Therefore, few-shot methods need a more explicit objective that regularizes the learning and avoids overfitting. One way of doing that is to induce a bias in the method. For instance, as objective the vector of an entity must be close to the average representation of their entity class and others should be far from each other. This is the base of prototypical methods.

Prototypical Networks were introduced by [

] adapted the idea to NER. These networks learn a token embedding such that each class is represented by a centroid vector, which is the average representation of the labeled entities of the class, and entities that are close to one of these centroids are classified as the same entity.

Concretely, each token token x is embedded with a pretrained language model f. Then the prototype of class ci is computed as:

For a given token, the model outputs its confidence on it belonging to class ci by using the Euclidian distance and the softmax function as follows:

The fine-tuning of parameters is done by maximizing the log-likelihood.

One simple idea is to leverage the information contained in the label name semantics to guide the model towards good prototypes, i.e good representations of each class. The representation of each token is compared to the vector representation of each label name. If the two vectors are similar, the token can be classified as an entity of this label. This idea was integrated into a model by [

As you can see in figure 3, each token class is represented by a small span indicating the kind of class and the position of the entity. These label names and the input tokens can be encoded by a BERT encoder. Then, for each contextualized embedding of a token, a dot product is computed with each label embedding and a softmax layer is applied to have the output scores. The training is done by backpropagating a classic cross-entropy loss.

While the idea seems interesting and results promised in the paper look good, we were not able to reproduce them.

A post-processing step can be added to improve performance. This step is not always reported in papers but can have a big impact on performance. The most common method is to use a Conditional Random Field (CRF) to estimate the transition probability from one label to another. If we suppose that yt is the class of token t and x is the input sentence of the model. Then we denote p(yt|x) as the output probability vector of word t and p(yt|yt-1) the probability given by the CRF of having label yt after yt-1. Then, the class of token t is given by solving the following problem:

, a simple and effective few-shot learning idea for NER that compares the representation of each entity to the k nearest neighbors to get the probabilities p(yt|x) and adds this CRF to improve results.

Meta-learning refers to a series of methods whose purpose is learning to learn. Concretely, in NER, this means that models are given several small NER tasks with different classes to train on such that, when they receive a new small dataset, they can train on it and get the best possible performance. One of the most commonly used algorithms in meta-learning is Model agnostic Meta-learning (MAML for short). MAML supposes that you have several small tasks, this means that you have a dataset D that is a set of several support and query dataset couples, 

. Each couple has its own set of classes. Then, the algorithm is as follows:

MAML was adapted to NER and tested on the main benchmark for few-shot NER, which is the FewNERD dataset. As of today, the state-of-the-art method for few-shot named entity recognition on this dataset was introduced by [

] and is called Decomposed Metal-Learning for NER. The idea of this method is to separate the NER task into two subtasks which are span prediction and entity classification. The goal of the first one is to detect the beginning, inside, and ending entity tokens and the second focuses on classifying each detected span. For each subtask, a model is trained following the MAML principle.  

The problem with this method is that it is quite long and we have to keep many copies of the parameters which can be memory-heavy in training. A labeled dataset with many different entity recognition tasks like FewNERD is also needed, which is quite hard to get and also adds some bias to the model. If the new task you want to train on has a very different text from the training data, generalization won’t be very good. 

In recent years, several few-shot learning methods have been adapted to NER with some promising results. We are still far from what humans can do and few-shot learning methods don’t generalize as well as traditional methods finetuned on many examples. Nevertheless, these methods are particularly well suited for pre-annotation because, in this setting, labeled data is scarce and reviewers can correct wrong labels.

 Most performant named entity recognition (NER) models require lots of labeled entity examples to be

Few-shot Learning methods for Named Entity Recognition

In Machine Learning, it is common to create a predictor that performs very well on average on the entire test dataset but very poorly on a specified subset (or slices) of data. 

These systematic errors are commonly referred to as hidden stratification and have caused huge failures in the development of Artificial Intelligence. For example, Amazon's facial recognition matching professional athletes as mugshots in the 2019 Super Bowl or Google identifying men wearing monocles as armed with a gun in April 2021. 

Identifying these subsets is the first step to improving the model robustness, either through robust optimization techniques or by adding relevant data. 

With structured data; it is usually quite simple to identify those slices, simply with visualization or data slicing methods ([

]). But with unstructured data  (images, text, audio), the problem becomes much more complex. Before exploring some recent methods developed to discover hidden stratification, we will try to understand how those slices of data can emerge and why it is difficult to detect them. Finally, we will see how we can leverage 

the dataset visualizations we created with Mapper

One of the main difficulties when trying to detect hidden stratification is the definition of the slices where the model underperforms. This is the consequence of the infinite possibilities in the choice of granularity. Let’s take, for example, a dataset where we want to classify between dogs and cats:

All stratifications are correct, this is why it is so difficult to define an algorithm to detect hidden stratification in a general framework and without a 

Nonetheless, we have seen in recent years some progress in this topic with the emergence of several Slice Discovery Methods (Multi-accuracy ([

]). They all rely on the same two steps: 

In the first Mapper article, we have already seen what data embeddings are and how we can build them with a pre-trained neural network. Some of the abovementioned methods go further by using cross-modal embeddings (DOMINO) or a robustly-trained model (BARLOW). 

PCA is the default dimension reduction method for most of the methods. GEORGE uses UMAP for dimension reduction and BARLOW a feature selection method based on mutual information gained (a statistical value that measures the importance of each feature for the classification task). Each method uses its own clustering algorithm, some choose only to highlight a few clusters that could match poor-performing slices (Spotlight, DOMINO) while others cluster the entire dataset to highlight potential sub-classes (GEORGE). 

Slice Discovery with Mapper’s Dataset Representation

built a dataset representation with Mapper

, we can see that we perfectly match the two steps of Slice Discovery Methods. Even better, instead of standard clustering methods, which only return lists of elements, we have a method that returns a graph, linking clusters to each other and providing a user-friendly visualization.

Topological Data Analysis’s ability to detect hidden stratification and subgroups has been known for more than a decade ([

]), and others have used Mapper to classify error types in a predictive process ([

] created “DC-Bench”, a data-centric benchmark consisting of a list of problems and associated models subject to hidden stratification. This benchmark can be used to compare Slice Discovery Methods based on their ability to detect hidden stratification. We use standard evaluation metrics such as precision (fraction of assets among the retrieved instances that actually belong to the slice to discover) or recall (fraction of assets that actually belong to the slice which are recovered by the Slice Discovery Method). 

We used it to compare the following methods: 

and where we consider the 5 (or 10) graph nodes where the model performs the worst.

Unfortunately, we could not recover the results presented for the DOMINO method. 

We compare the precision of all created clusters (meaning the percentage of value among the cluster that belong to the actual slice): 

We use precision as our metric because the purpose is to facilitate the detection of hidden stratification for the user. This means that, when looking at the few nodes constructed by Mapper where the model underperforms, the user can easily identify a coherent subset corresponding to a systematic error of the model. 

If we look at the example of the previous article: 

If I look inside node 4, where my model performs poorly (on average, the model projects a 17% probability of belonging to the correct class to assets inside that node), we see “T-shirt/Top” with complex patterns, which is a systematic error of the model. To solve this issue, we can train our model to recognize Tops with complex patterns or add pictures matching this criterion.

Hidden stratifications are the source of important failures in Machine Learning. Their automatic discovery is a hot topic in the Data-Centric paradigm of Artificial Intelligence.

With Mapper’s interactive dataset representation, we can help data scientists identify hidden stratification. While still relying on a human in the loop, this method is both powerful and user-friendly, thanks to graph visualization. 

Gurjeet Kaur Chatar Singh, Facundo Mémoli, and Gunnar E. Carlsson. Topological methods for the analysis of high dimensional data sets and 3d object recognition. 2007. URL 

http://dx.doi.org/10.2312/SPBG/SPBG07/091-100

Neoklis Polyzotis, Steven Whang, Tim Klas Kraska, and Yeounoh Chung. Slice finder: Automated data slicing for model validation. 2019. URL 

Michael P. Kim, Amirata Ghorbani, and James Y. Zou. Multiaccuracy: Black-box post-processing for fairness in classification. CoRR, abs/1805.12317, 2018. URL 

Nimit Sharad Sohoni, Jared A. Dunnmon, Geoffrey Angus, Albert Gu, and Christopher R ́e. No subclass left behind: Fine-grained robustness in coarse-grained classification problems. CoRR, abs/2011.12945, 2020. URL 

Greg d’Eon, Jason d’Eon, James R. Wright, and Kevin Leyton-Brown. The spotlight: A general method for discovering systematic errors in deep learning models. CoRR, abs/2107.00758, 2021. URL 

Sahil Singla, Besmira Nushi, Shital Shah, Ece Kamar, and Eric Horvitz. Understanding failures of deep networks via robust feature extraction. 2021. URL 

Monica Nicolau, Arnold J. Levine, and Gunnar Carlsson. Topology-based data analysis identifies a subgroup of breast cancers with a unique mutational profile and excellent survival. Proceedings of the National Academy of Sciences, 108(17):7265–7270, 2011.URL 

https://www.pnas.org/doi/abs/10.1073/pnas.1102826108

Leo S. Carlsson, Mikael Vejdemo-Johansson, Gunnar Carlsson, and P ̈ar G. J ̈onsson. Fibers of failure: Classifying errors in predictive processes. Algorithms, 13(6), 2020. ISSN 1999-4893. URL 

Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Large-scale learning of general visual representations for transfer. CoRR, abs/1912.11370, 2019. URL 

Discover fairness issues in classification with Topological Analysis of the Dataset

Today, AI plays an increasing role in patient care. Healthcare providers, researchers and data scientists have teamed up to advance 

 (ML) and medical diagnoses. Given the aging population and expansion of personalized gene therapies, ML models look likely to play a vital role in future healthcare provision.

Deep machine learning and medical diagnosis examples

Computer vision and ML can enhance the valuable microscope work traditionally carried out by pathologists. 

 is a common thread in diagnostic applications, which evaluate physiological data, environmental influences and genetic factors.

Thankfully, a large medical dataset is available to clinicians, ranging from symptoms to test results and image files. Detailed analysis of this resource results in a better understanding of biological mechanisms and risk factors. Next, we consider some examples in detail.

How is machine learning used in medical diagnosis?

Artificial intelligence helps with decision-making, workflow management and the automation of tasks. Healthcare professionals use the latest machine learning in triage to flag abnormalities and prioritize life-threatening cases. Similarly, AI supports physicians in diagnosing cardiac arrhythmia, predicting stroke outcomes and managing chronic diseases.

AI is already facilitating diagnosis, drug development, personalized treatments and gene editing. Increasingly, these insightful tools are revolutionizing healthcare, thanks to causal reasoning techniques in ML algorithms. Earlier models used only correlations between symptoms and the most likely cause(s).

, Dr. Jonathan Richens and colleagues outlined the latest approach whereby algorithms work through the possibilities of patients' symptoms being due to various conditions. In research, experts gave AI the ability to abstract alternate realities and consider whether one or more symptoms could be present if the patient had a different disease. These reiterations meant ML algorithms scored higher than seven in ten doctors on written test cases.

Stanford University researchers trained an algorithm to diagnose skin cancer using 

. In a convolutional neural network (CNN), an ML algorithm learned to detect cancers and melanoma using large datasets containing 130,000 images of skin lesions and more than 2,000 different diseases.

Physicians diagnose around 5.4 million cases of this type in the US every year. Early detection improves the prognosis for survival but – as statistics show – delays increase mortality rates.

Visual examinations are crucial in dermatology diagnosis. First, a dermatologist inspects the lesion(s) of interest with a dermatoscope, i.e., a handheld microscope. Then, a biopsy is necessary if the tissue appears cancerous or the evaluation is inconclusive.

Researchers tested Stanford's deep learning algorithm against twenty-one certified specialists who examined 370 images. The machine's predictions had the same precision as all the dermatologists in deciding the best course of action*. Nonetheless, the research team highlighted the need for rigorous testing before integrating the ML model into a live environment.

Historically, pathologists diagnose certain diseases manually by observing microscope images. This method has changed little in more than a century.

Researchers from Beth Israel Deaconess Medical Center and Harvard Medical School recently labeled hundreds of scan results to speed up diagnoses and boost accuracy. They highlighted cancerous and noncancerous cells to form the training data for deep learning. The ML algorithm achieved 92 percent diagnostic accuracy, marginally lower than the human rate of 96 percent. In combination, however, algorithm predictions and human diagnoses boosted accuracy to 99.5 percent.

In the UK, researchers at the Queen Mary University of London used AI to analyze blood samples from rheumatoid arthritis patients and predict their response to treatment. Some anti-rheumatic medications do not benefit as many as half of the patients prescribed them. In contrast, the same drugs could regulate inflammation well in others.

 article, Professor Jesmond Dalli noted that many patients' conditions had not responded well to their initial prescription. These patients suffered unnecessary side effects for six months while assessing the treatment's effectiveness (or lack thereof) before changing. Their disease had worsened before trying a different treatment that was more likely to work.

Following changes, patients' therapy became more focused and personalized. Doctors used the new biomarkers to foretell the effectiveness of different disease-modifying anti-rheumatic drugs.

Using AI to reduce the risk of heart attacks

A separate UK-based team developed an ML model that spots red flags for coronary incidents. University of Oxford professor Charalambos Antoniades and colleagues discovered a new biomarker called the fat radiomic profile.

The new medical technique flags anomalies in the perivascular space around the heart. In the 

, the medics explained how the app identifies genes associated with inflammation, scarring and other changes. The dataset comprised test biopsies from 167 cardiac operations.

According to Professor Antoniades, the absence of noticeable narrowing in a coronary artery does not necessarily exclude the possibility of heart attacks. Therefore, the new diagnosis used AI to search carefully for disease characteristics around blood vessels, thus enabling timely prevention.

The US Institute of Medicine reported that diagnostic errors contribute to approximately 10 percent of patient deaths and a similar proportion of complications in treatment. Although the performance of medical professionals was not necessarily in question, a significant number of errors arose from communication failure, non-integrated healthcare IT and computers that did not support proper procedure.

As well as addressing the above issues, AI supports clinical staff and helps reduce burnout due to overwork and exhaustion. In addition, the latest healthcare computer systems enable streamlined methods to save time. For instance, radiologists can manage dozens of images with greater ease. As a result, the capacity and accuracy of diagnosis have increased; specialists can quickly check and flag scanner images to deal with urgent cases more efficiently.

Machine learning also benefits drug development and trials, clinical research and robotic surgery. In the latter, automatic suturing could shorten operations and reduce surgeon fatigue.

Of course, continued rigorous testing of these applications will be necessary to 

 their utility. At the same time, some clinicians are unconvinced. They envisage challenges in integrating AI in clinical practices without undermining their expertise. Therefore, education and adaptation of healthcare systems will be instrumental in rolling out and adopting these new technologies.

For hospital and clinic managers who want to incorporate ML into their patient care and business administration systems, expert help is at hand. The precise steps vary in line with the tasks under automation, from research and clinical trials to speeding up diagnosis and managing disease. Nonetheless, a common thread is machine model training which involves setting up a training dataset and labeling it – also known as data annotation.

To ensure accurate data annotation, we offer 

 in implementing its leading-edge healthcare solutions.

AI is, in effect, a second pair of eyes. We have seen an overview of its positive contribution to medical diagnosis, patient treatment and care. Also, ML supports medical practitioners and clinics by alleviating undue workload pressure and maximizing efficiency.

*Journal Nature: Dermatologist-level classification of skin cancer with deep neural networks.

Machine Learning and Medical Diagnosis: An Introduction to How AI Improves Disease Detection

We live in a digital world where businesses depend on data to effectively run business operations, such as delivering products and services to their clients. Unfortunately, data exists in various formats and is often trapped in documents, necessitating time-consuming and error-prone manual extraction and conversion methods.

Some enterprises have successfully automated and simplified these processes using traditional solutions. But that doesn't change the fact that these outdated techniques and systems pose a significant threat to customer-driven innovations. Luckily, advancement in technology has given birth to various innovative solutions, such as Intelligent Document processing (IDP), that offer many benefits in document processing.

In the most basic dictionary definition, Intelligent Document Processing is a software solution that employs various technologies to automate document processing. Basically, an IDP solution extracts unstructured & semi-structured data and converts it into structured data.

Since it does not have a pre-defined model or structure, unstructured data can be challenging to store or index. Consequently, the unclear structure poses a myriad of data extraction challenges for users. Luckily, a reliable, intelligent document processing solution can automate data processing, making information extraction easier.

IDP solutions share a few similarities with Intelligent Character Recognition (ICR) and Optical Character Recognition (OCR), which are used to capture text from scanned images. However, IDP is slightly different from these two solutions since it captures, extracts, indexes and processes the data using various intelligent technologies.

Modern businesses use IDP solutions for wide range of use-cases. These include automation of:

How Does Intelligent Document Processing Work?

As we already mentioned, Intelligent Document Processing uses several artificial intelligence technologies to automate document processing. These technologies include:

Data processing using an IDP solution follows several stages consisting of various components. Also, each of the technologies mentioned above features in different stages and serves specific functions.

Stages Involved in Intelligent Document Processing

As the name suggests, the pre-processing stage involves preparing documents for processing. It involves maximizing the quality of documents using methods such as de-skewing, binarization, and noise reduction.

Once IDP maximizes the document's quality, it proceeds to the next step. In this stage, the solution utilizes computer vision to identify the contents of the documents (including text, images, and charts). It can also leverage traditional technologies such as ICR and OCR software to extract text from the document. Essentially, the image processing stage is where the documents undergo a digital transformation, which primes them for machine learning.

Stage 3: Data Classification and Extraction

Once the digital twin of the document is complete, the IDP solution proceeds to data classification and extraction. In this stage, the solution employs the help of Natural Language Processing (NLP) to understand the meaning of the text. 

After identifying the contents of each document, IDP uses machine learning to place each document in a class. For example, a business report might include accounting documents, such as bank statements, invoices, receipts, etc. Using a classification engine, an intelligent document processing solution parses out the components of the report, indexes and forwards them to their respective destinations.

The integrity and accuracy of processed data is paramount. For that reason, intelligent document processing solutions have a data validation stage where they validate the quality of data extracted from the documents. These solutions leverage HITL machine learning framework technology that allows humans to inspect and correct any mismatching data.

After the data is validated, the next step is integration into various systems and workflows across the company.

We help customers to build AI that matters across industries

They trust us for their data-centric journey

Get Started