What is the Definition of Machine Learning?

Here, we explain what machine learning (ML) is and what it involves. You will also find an overview of its beginnings, the characteristics of different types and an

 introduction to its challenges. Finally, we discuss the likely rewards that today's forward-thinking companies can reap from artificial intelligence and ML. Please read on to discover more.

(AI), ML algorithms analyse situations and predict results. The discipline employs tested computer science principles inspired by what humans learn. Let's delve deeper into the fascinating world of Machine Learning! 

ML dates back more than six decades to the early development of a computer program to play drafts or checkers. In 1959, A. L. Samuel published details* of his research into automating the board game. This first paper became a landmark in ML after the computer won a game of draughts against a human. In his paper, Samuel described early notions of machine learning. He envisaged that programming computers to learn from experience would eventually eliminate detailed coding efforts. During the 1960s, computer programmers and researchers enhanced this early model. Then, of course, the following decades saw the introduction of ever more powerful chess algorithms. Nowadays, applications and automated systems are becoming increasingly rapid and accurate in their performance.

ML is a subset of AI and it features a number of algorithms, ranging from statistical modeling, support vector machines, ensemble methods and artificial neural networks. In artificial neural networks, each node has threshold settings that influence whether it sends or ignores the information in response to inputs. Training and – to a lesser extent – validating the model involves adjusting the weightings between these neurons. Linear regression is another approach that is one of the most popular. It uses the principles of statistics and mathematical equations for predictive analysis, especially with fluctuating variables such as age, salary, sales and price. In essence, the technique interpolates and looks for the line of best fit among a series of dots – each a data item – on an X-Y graph. Regression techniques also help us understand how dependent variables change in response to primary (independent) variable fluctuations, assuming that other values remain the same. Another approach, 

, is used for classifying true or false situations, i.e. those that result in a yes/no condition or a Boolean output of 0 or 1. Deep learning networks are artificial neural networks that have four or more layers, while a basic neural network would have just two or three.

Machine Learning vs. Deep Learning vs. Neural Networks

Although some people use the terms machine learning and deep learning interchangeably, there is a distinction between them. 

 uses neural networks to identify features in the data without the necessity for human intervention; it is a subset of ML. Deep learning can gain insights from labeled datasets. Unsupervised deep learning does not require labeling or tagging to function. Instead, it can process unstructured data autonomously and determine which characteristics distinguish different categories of records from each other.

, software engineers or developers use a labeled data set to orient the machine learning model, for example, a neural network during training, validation, and testing. The labels or tags identify characteristics or properties that enable the correct classification of each record. Then, diagnostic algorithms use cross-validation techniques to ensure accuracy. The machine must receive sufficient data without causing bias due to overfitting, which results from skewed inputs or too much information of one type.

Unsupervised learning analyses clusters of information to detect hidden patterns and find groupings. This method is beneficial when we want to discover the differences and similarities, such as in customer segmentation exercises or to develop cross-selling strategies in business.

Semi-supervised learning combines the two previous methods by using a degree of labeling or tagging to assist classification. In addition, it supports the extraction of features from a more extensive import that is unlabelled. This hybrid approach is advantageous when a lack of labeled data or training in a supervised learning algorithm would be too expensive.

 will enable you to get a more detailed understanding of the field.

Reinforcement ML algorithms interact with their set environment to optimize rewards. As a result of each step in this trial and error, the ML model receives positive signals for correct actions – or negative cues for mistakes. The method enables machines to maximize performance or determine the ideal behavior in specific contexts. Data scientists use reinforcement learning to teach neural networks multi-step processes for which defined rules exist. Instead of data, the human input involves positive or negative feedback cues. However, the 

themselves decide what steps to take. Reinforcement learning is helpful in operations research, swarm intelligence, and simulation-based tasks to optimize resource usage. Under this model, software agents taking actions based on cumulative reward is comparable to human behavior and motivation in a broad range of situations. Analogies range from decision-making based on essential survival priorities to maximizing retail marketing returns from in-store footfall patterns. In business, reinforcement learning helps companies plan the optimal allocation of finite resources. Conversely, reinforcement learning models have learned to compete in various video games in the leisure arena.

Reinforcement ML is a particularly suitable technique in gaming, robotic tasks, and controlling autonomous vehicles. Supervised learning is helpful in translation engines with incomplete dictionaries. In these cases, algorithms may learn to translate written language (or speech to text) by analyzing context, checking grammar, and interpolating meaning. Similarly, semi-supervised learning can be valuable in analyzing statistics and preventing fraud. It is usually possible to teach a model to identify suspicious cases using relatively few positive examples. In this application, 

analyze similarities, looking for patterns and exceptions in banking and commercial transactions. Some of the most frequently seen examples of ML include:

Implementing ML enables business development and marketing teams to understand and appreciate customers' needs in greater depth. It also means they can monitor customer behavior and create fruitful sales initiatives for products and services in different industries. Similarly, in CRM (customer relationship management), the latest software reads incoming company emails and directs them accordingly. In most cities, smartphone applications for online private hire and taxi services are ubiquitous. These popular ride services use AI algorithms to match drivers and vehicle locations to passengers' requests.

Although AI and ML have several advantages, a few factors require consideration before embarking on a project. Firstly, the cost of setting up the necessary software infrastructure can be relatively high. In addition, most projects call for the services of data scientists and skilled researchers. Also, the process may well require allocating internal resources and working time, particularly with 

Alternatively, some companies outsource this time-consuming task to an expert service supplier. The source(s), format, and quality of data imports are essential to project success. For these reasons, project planning and execution during the learning and validation phases must avoid ML bias. The latter tends to occur through overfitting, i.e. tuning the machine learning model too heavily on a subset of data that is too different from the “real-world” data. On the other hand, insufficient data is also likely to cause inaccurate output. So, we can see that expertise and experience are valuable from the design stage.

Thanks to advances in processing power and storage capacity, technological developments are set to continue. Already, automated human resources information systems can filter through applications and identify interview candidates based primarily on keyword scanning. As well as current projects such as autonomous cars and surprisingly accurate video streaming recommendations, the technology will surely expand to other business areas and questions. In robotics, which relates to the physical world, reinforcement learning can teach machines to perform routine, repetitive tasks. Apart from manufacturing, we can expect more comprehensive automation in service transactions from food waiters to travel and customer inquiries.

As the use of ML increases in automated decision-making, so will the demand for training and validation data. However, these imports often contain potentially sensitive information, often in the medical and finance domains. Data protection legislation, including GDPR, requires the safeguarding of personal data. Article 35 of that directive compels organizations to analyze, identify and minimize data protection risks for every algorithm and project. To address this critical need, open-source tools such as ML privacy meters enable developers to quantify privacy risks. The tool highlights training data records that could leak through model parameters or algorithm predictions.

Algorithms trained on similar data are liable to result in unreliable output that does not reflect real-world situations. At best, these flawed systems are irrelevant or inaccurate. At worst, they could fail or be discriminatory by excluding specific customer profiles or populations.

Currently, AI models need extensive training to optimize them to perform a single task. In the future, researchers look set to produce more flexible models. It may become feasible to develop techniques that enable a machine to retain learning from the context of one or more previous tasks and apply it to new jobs. However, concerns have arisen about the idea of AI surpassing human intelligence. While this superintelligence is unlikely to occur soon, some commentators have indicated it might happen within a few decades. Consequently, they have suggested keeping any superior intellect on a short leash if it outperforms humans in creativity, wisdom, and social skills.

Other issues include questions such as: to whom would the occupant(s) of a driverless car be responsible for an accident? Recent advances in motion systems, sensor accuracy, and processor speed mean that robots are becoming more nimble and life-like. Four-legged dog-like machines now form part of explosive ordnance disposal equipment. However, the possible military repurposing of these defensive robots into highly agile machines that can run, pounce, and attack is a concern for those advocating pacific uses. By last year (2021), news stories** featured the US Army testing armed robots with sniper rifles.

Today, AI is a rapidly growing field. In ML, we use statistical methods to classify data, make predictions and provide valuable insights. Deep learning powers the most advanced applications in AI. Enterprise technology vendors such as Microsoft, Google, IBM and Amazon offer various platform services and competition look set to intensify. However, 

to iterate smoothly on your training data, train your AI faster, and ship turnkey projects ready for production. With Kili, you can better manage training data, whether images, video, text, voice, or PDF files. The benefits of this leading solution include rapid annotation, quick collaboration, intuitive quality control, and straightforward data management. The software is available either as a cloud service or local installation to match all requirements. Collaboratively, Kili eases the alignment between SMEs and ML experts. It enables developers to build machine learning algorithms using the latest techniques and accessible languages, including Python. If you are a CTO, technical CXO, or business decision-maker and would like to discuss your requirements, please go 

. Our team of approachable experts will be delighted to answer any questions you may have.

Discover more about the mechanics of Machine Learning through our resources:

Machine Learning: Defined and Explained

Get Started

Data labeling is the task of systematically recognizing and identifying specific objects within raw digital data, such as 

), thereby “tagging” them with digital labels that enable machine learning (ML) models to create accurate forecasts and assessments. 

Here is some further reading material on other forms of data 

. These pertain more to the context of natural language processing.

Data labeling and data annotation techniques are characteristic of the initialization stage when developing a machine learning model. It requires the identification of raw data, and then adding one or more labels to that data, to specify its context for modeling. This preprocessing stage permits the machine learning model to assemble accurate predictions.

For example, digital data annotation can permit autonomous automobiles to stop when approaching pedestrian crossings, digital assistants can recognize certain human voices, and digital security cameras can notice and witness strange or suspicious conduct.

 has made significant advances within the last twenty years or so. This has been attributed to concurrent and contemporary improvements in computing processing power and the major breakthroughs achieved in deep learning research. Another key catalyst has been the quantity of digital data that has been collected and stockpiled. These advanced algorithms have progressed rapidly, and their need for digital training data has kept pace too.

A key concept required for successful machine learning outcomes is the technique called supervised learning. Supervised learning is a machine learning approach defined by utilizing labeled datasets. These datasets are designed, over time, to train or “supervise” algorithms to correctly classify data, thereby predicting accurate outcomes. The machine learning model can measure its accuracy by processing the datasets and, therefore, self-learn over time.

The collection of a significant amount of digital data – images, video stills, videos, audio files, transcribed texts, etc. – is typically the first step in the process. A large and mixed quantity of digital data will guarantee more accurate computerized results in the long run than a diminutive amount of digital data can provide.

Digital data tagging consists of human enforced data labelers who identify those entities amongst the unlabeled data while operating on a data labeling training platform. For example, human data labelers might be tasked with demarcating whether a digital image comprises people (or not) or how to track a tennis ball during a tennis match within a video.

The labeled or annotated data must be instructive and accurate to nurture high-performing ML, business models. Therefore, an internal and trusted quality assurance process is mandated to review the accuracy of the digitally labeled data. Otherwise, there will be a high risk that the ML business model will fail to perform satisfactorily.

What Factors Affect the Quality of Data Labeling?

With data labeling, there is a key requirement for basic domain knowledge and contextual understanding within the labor workforce. These skills are mandatory to be able to construct high-quality, structured digital 

 business models. Data labelers will label their data with a higher level of quality when they understand the reasoning and context of the data they are digitally annotating.

You can learn more about datasets through these useful links: 

How to create a dataset for Machine Learning

ML business modeling is an iterative process. Data labeling will naturally evolve as the ML business models' constructive testing and quality validation develop, and key learnings are adopted from their outcomes.

Digital data labeling units must have the ability to be flexible to merge changes that adjust to meet the end users' changing needs or products. A robust, yet flexible, digital data labeling team should be able to react positively to data volume changes, increases with task complexity, and task duration shifts. The more adaptive the labeling team can be, the more successful the ML business projects will be.

Positive and direct communication within the digital data labeling team is vital. Any practical and closed feedback loop is an immaculate means to establish dedicated communication and cooperation between the ML project team and the digital data labelers. In addition, data labelers are always advised to share their key learnings as they digitally label the data, so these valuable insights can be adapted to modify the approach if required.

Some Common (and Important) Missteps to Avoid During The Data Labeling Process

Once a data labeling team is established, performs its duties, and commences delivering valuable results, it can easily get carried away when deep diving into data labeling for any ML business project. Instead, one needs to be mindful of the missteps and pitfalls that may derail any ML project – and certain principles or rules must be adhered to when growing the digital data labeling enterprise.

1. Punctuation Issues, Excess Whitespace, and Case Sensitivity Rules

When performing data labeling, it is vital to ensure that the tag finalization process is merely a continuation of the format process that has been created  – and agreed upon – beforehand. Any new tags that are required to be added must be previously communicated to the entire team. When annotating text, for example, whitespace and other punctuation issues can be problematic. Such differences can generally be rectified using an assortment of computer scripting tools that are available on the market.

Any digital labeling exercise will encounter various challenges to overcome along the data annotation journey. However, occasionally some challenges that are encountered are a symptom of a more intrinsic issue that needs to be dealt with. For example, within the medical imaging industry, there is a highly-skilled requirement for the data labeling workforce to have a base level of medical knowledge and practical experience when viewing intricate images resulting from X-rays, MRI, and CT scans.

Further issues may arise when dealing with the scenario of inverted or nested data tagging. Let’s use a text document as an example: an initial tagged sentence or phrase may be correct since the digital tags were used according to the order in which they appeared within the sentence. But what about the scenario where the exact same words or phrases occur but in a different order, yet still with the same meaning? We could end up with a totally different order of digital tags, which therefore will attach a totally different meaning to what are essentially the same words, but in a different order. When using nested labels, all of the resources that are digitally labeling the ML dataset must operate in the same format. This issue can be extrapolated to other data types, such as images and videos.

3. Adding Any New Labels to Your Project Mid-Term

Halfway through a digital labeling undertaking, your team may realize that there will be a need for a new digital label that is currently not present in the agreed-upon master labeling list. In one initial undertaking, the team might simply add a new required data label to the master list and thereby inform the whole team to now use this new digital label when required.

But this process should be highly discouraged, and the simple reason is that the labeled data the team has already marked would have to be re-examined to see if the new label suddenly fits into the already label-processed digital data. It is best practice to always compile and agree upon the list of digital labels before commencing any labeling endeavor. If a new data label is devised as a required mid-project, it is ideal to take note of this occurrence and then review – as a team – how the data label was missed after the data labeling exercise is completed, and how this discovery can be prevented from re-occurring beforehand.

Acquiring high-quality digitally labeled data is an expansion barrier that becomes more significant when contemplating the handling of complex ML models. Yet, there are many strategies and approaches to improve the efficacy and precision of the data labeling process:

AI is revolutionizing the course on how we perform certain repetitive functions, and enterprises that have adopted such business models are gaining the benefits. The technological opportunities AI can deliver are potentially inexhaustible and will assist diverse industries to become smarter: from medicine to agriculture, to recreation and sports. Data digital annotation is the first stage towards such innovation.

Data Labelling: The Ultimate Guide

What are the Benefits of Intelligent Data Extraction?

Intelligent data extraction can help optimize workflows and boost efficiency for many different companies across industries. By taking advantage of automation, organizations not only free up their employees to work on other areas, but they can extract critical information far faster and with greater accuracy than by the equivalent manual processes.

As an example, intelligent data extraction is widely used in industries such as healthcare. Within healthcare, many important systems may still use paper-based documents. Without automation systems in place, these documents may need to be manually transcribed, which takes time and potentially introduces errors into the system in the form of transcription errors and other mistakes.

 automates much of the process, increasing workflow efficiency while also reducing the error rate. Optical character recognition (OCR) technology is used to scan and extract text from documents, which can then automatically identify and extract the key data needed from within the document. In a healthcare system, this can be used to automatically identify prescriptions, patient transfer forms, physician notes and more. This works whether the text is handwritten or typed, saving both time and money over manual processes.

Why is Data Extraction so Difficult Without AI?

Artificial intelligence (AI) is not necessary in order to automate data extraction. It’s certainly possible to run ‘dumb’ data extraction across documents, but there are a number of issues that present a significant challenge in doing so. Without AI, dumb processes are generally fairly rigid - they depend on a consistent data structure or look for specific, hard-coded keywords to extract. Unstructured data presents a challenge, as do synonyms or misspellings. 

Due to these inconsistencies, errors are much more frequent and much more time needs to be dedicated to validation in order to catch these issues before the data is propagated into the wider system. The 

 of paper-based systems may also require training staff on new processes, which can be completely bypassed with automated systems.

What is the Best Way to Automate Data Extraction?

Intelligent data extraction represents a digital transformation in how documents are consumed by businesses. There are a number of solutions to automate this process, but one such way is to combine OCR technology with text classification and named entity recognition. These technologies allow for paper documents to be scanned and have the content understood in a digital format. Both text classification and named entity recognition are forms of 

 and are used to recognize the type of document it is (e.g an invoice or purchase order) and then be able to automatically pull the key information (such as a list of the items ordered and their price). This can then be fed into other automated systems to quickly and efficiently handle paper-based processes in an automated manner.

How is Automated Data Extraction Being Used?

Many different industries are already using traditional OCR technology and automated data extraction to deal with document analysis, data extraction, validation, processing, and more wide-ranging functions. These include banking and finance, healthcare, legal professionals and plenty more. Below are some concrete real-world use cases of how businesses across industries are utilizing this technology to improve operational efficiency and cut costs on manual document processing.

While much of banking and finance has been moving towards paperless systems, there are nevertheless still a lot of systems in place in finance that make use of paper-based documents. Invoices, purchase orders, loan and mortgage application forms, as well as accompanying documents, are all examples of paper-based documents that must be processed every day by banks at scale. In order to meet this demand in a cost-efficient manner, automated data extraction is used to identify and categorize documents, as well as extract key data from them that can be digitized and used within a bank’s wider electronic systems.

The legal services industry is incredibly document-driven, with many of these still being printed on paper. Everything from depositions to court summons, warrants, litigation filings, court orders, and contracts are printed on paper, but all need to be kept track of. This is a monumental amount of paperwork generated on a daily basis that must be accounted for. Intelligent data extraction represents a 

 in how this mountain of paperwork is sorted, analyzed, and stored. 

Supply chain management is another industry that generates large volumes of paperwork, with much of it being unstructured or sorted into many different file formats. Everything from invoices to purchase orders and customs declaration forms needs to be handled, with different clients and countries providing their own formats for these documents. Intelligent data extraction is used to quickly and easily categorize and process these documents without requiring the manual homogenization of many different formats for invoices or purchase orders.

Why You Should Use Intelligent Data Extraction

How Can Your Business Utilize AI Document Analysis?

Data is the lifeblood of many businesses today, with many companies handling millions of documents on a regular basis. AI document analysis helps businesses to consume, extract, and process important information from this large volume of data in an efficient and practical manner. Read on to learn what AI document analysis is and how it is leveraged by modern businesses as part of their regular workflow.

 technology enables companies to process large volumes of documents that would otherwise require significant time and resources to process manually. As an example, in banking millions of invoices, contracts, mortgage applications and other forms of paperwork are received on a daily basis. To process all of these manually would take significant manpower, be prone to error, and represent a significant expense in terms of both time and budget. Instead, AI is used for 

, document insights, and processing content. This increases operational efficiency and allows a business to provide users with a much faster and smoother service.

AI document analysis leverages machine learning (ML) and natural language processing (NLP) to perform processing tasks, provide insights, and help automate workflows involving physical documents. Firstly, optical character recognition (OCR) technology is used to identify text on a scanned document, whether it be handwriting or printed text. Early-stage heuristic functions are then applied to quickly determine what type of document it is so that the correct models can be applied to process it. For example, the layout of mortgage application forms likely follow a consistent format. Identifying that format allows the system to slot it into the correct workflow without having to process the entire document first.

What Kind of Industries Benefit from AI-Based Document Analysis?

There are many different use cases for AI based document analysis. 

 technology allows companies to perform document analysis and process documents as part of automated workflows, slotting in to existing systems and models. The number of industries that stand to benefit from this is virtually unlimited - any business or industry that deals with documents in some form stands to benefit from the features provided by such systems. These industries include:

Many of these industries deal with large volumes of documents on a daily basis and each may have separate requirements and needs for handling and processing these documents. Fortunately, a sufficiently mature platform can process thousands to millions of files for both simple and complex use cases.

What Value does Automating Document Analysis Bring? 

Much like there are multiple use cases for this technology, there are also multiple benefits that it brings to companies that utilize it. Listed below are just some of the benefits automating document analysis brings to companies.

Manually processing documents is a time and manpower-intensive task. Not only that, it is unavoidable that mistakes will be made along the way, which takes additional time and manpower to track down and correct. By automating these tasks, employees are freed up to perform tasks where they are better suited than AI, such as in creative endeavors or those requiring client-facing interaction, rather than repetitive tasks.

In addition to the regular processing of legitimate documents, AI and ML can be trained to identify fraud. This is particularly useful in insurance and finance, where 

 can be used in conjunction with document analysis to identify and flag likely fraudulent features, saving potentially millions of dollars a year otherwise lost to fraud.

Standardizing Document Processing and Storage

Manually sorting and storing documents opens up the possibility of irregularities, which reduces efficiency. Time must be taken to train employees in the correct business processes for handling documents and any mistakes or errors can lead to documents going missing. By automating this process as part of document analysis, the business procedures put in place are codified and automated, ensuring consistent document handling and storage every time.

Leveraging Patterns in Data to Develop Key Insights

AI and ML processes can be used across large data sets to discover key patterns and insights that are otherwise very difficult, or not possible, to discover when looking at documents individually. AI document analysis allows you to contextualize data that is spread across thousands to millions of documents and can help to evaluate market trends, such as seasonal variations in retail sales as an example.

Your Guide To AI Document Analysis

Your Guide to AI Document Analysis

Named Entity Recognition (NER) refers to methods and processes of identifying entities in text and applying labels to them that can be used for further analysis or processing. As a simple example, NER may recognize the name ‘Netflix’ in a document and apply the label ‘company’ to it. NER is extremely flexible and many different kinds of entities may be defined, identified, and sorted into categories, which means that it is capable of entity recognition across people, places, companies, periods of time, and more. 

NER leverages natural language processing (NLP) to identify and tag entities based on defined parameters. 

 pulls out and tags important entities identified within a document. This can help to quickly and automatically identify the most important pieces of information within a document. As an example, NER can be used with invoices to automatically identify the account ID, the shipping and billing addresses, and invoice amounts. This can then be integrated into a company’s electronic payments system to allow for the automatic processing and payment of invoices, even if the initial invoice is received on paper. 

Text classification has some similarities to NER but fulfills a different role. Text classification looks at a text holistically to make a judgement about its content. In contrast to 

 which extracts specified entities within a document, text classification looks at characteristics of the text as a whole to draw conclusions about things such as text sentiment, the language of the text or the text topic.

There are several different methods by which text classification can categorize a 

Sentiment analysis refers to gauging the sentiment of a piece of text, such as whether the author is expressing a positive or negative opinion within the text. The use of sentiment analysis is very popular within social media for understanding how consumers feel about a given topic or brand. Sentiment analysis analyzes the language used within a body of text to determine its sentiment. It can be either positive or negative or more sharp classes, specific to the use case.

Text classification can be used to simply detect the language of a corpus of text. This has many uses, such as automatically routing support queries across different languages to the correct department or for offering automatic language translation services if a body of text does not match the user’s language locale settings.

To stick with the example of a support query, topic classification is used to automatically identify the nature of a support query without relying on the customer to correctly tag the request. By identifying certain parameters or phrases used within the support request, text classification can identify whether a customer support query relates to billing or troubleshooting, for example, and then automatically route the support request to the most appropriate department. 

How are both Concepts used to Automate Document Processing?

NER and text classification both have distinct roles to play in automating document processing but they can be used together to effectively and automatically analyze, categorize and process a wide range of documents. A good example of both of these technologies coming together is when handling customer support queries.

Text classification can be used to determine the language, topic and sentiment of the support query. The language identification is used so that the support query is routed to the correct support team members who speak the same language. Then, topic classification can be used to judge the type of query: is it one about billing or invoices? Is it a technical support request, or a complaint? This helps customer queries to be routed to the correct department automatically. Lastly, sentiment analysis can be used to make a judgement about how the customer feels, such as whether they are angry or upset, and can help assign an urgency to the query.

Named entity recognition can be used to extract important information from supporting documents related to the query. For example, if a customer is getting in touch with a bank about a loan application, then they may need to supply supporting documentation, such as a proof of address. Named entity recognition can be used to quickly and automatically verify whether the document provided contains an address and whether the name of the document matches the name of the customer within the bank’s systems. 

Together, these concepts help to reduce the amount of manual work an organization must do in order to support requests at scale, particularly those involving supplementary text that may come in a variety of formats and structures.

How have these Concepts Evolved and what is their Future?

The ability to identify and classify information into discrete categories has been around for a long time and does not require machine learning or deep learning to achieve satisfactory results. The earliest forms of NER and text classification were done by defining rule-based sets, often based on the presence of keywords or set 

. Data extraction was done by cycling through this list of defined entities rather than relying on an 

. These approaches suffered from relatively low precision as a result. Spelling mistakes, synonyms or foreign languages would all result in failing to extract important information and often required human validation.

The introduction of deep learning and AI to the field has enabled technology to overcome these problems and allows for more abstract concepts to be identified within text rather than hard-coded words or phrases. As we look towards the future, these technologies will evolve from information and entity extraction to knowledge extraction. Rather than simply understand the sentiment or topic of a piece of text, these systems will eventually be able to incorporate the knowledge contained within a piece of text into their wider system. This could mean the automatic understanding, processing, and resolution of a customer support request without the need for human intervention at any stage. 

Another interesting future development for this technology is ontology learning. This means being able to identify and understand entities within a piece of text and to also be able to identify and extract relationships between those entities within a given domain without requiring human intervention. This would allow for a richer and deeper understanding of a text to be developed from natural language text without specific training.

Understanding Named Entity Recognition & Text Classification

PDF text recognition can automatically recognize and extract text from a PDF document and present it in a readily available text format. Rather than having to manually transcribe a PDF document, instead OCR (

) technology is used to automatically identify text elements and convert them into usable text that can be searched and copied. Read on to learn how this technology works and why it is useful.

PDF text recognition utilizes OCR technology to identify characters in a 

. OCR works by analyzing the patterns of light and dark pixels that make up the defined characteristics of each character and then applies these patterns to known rulesets in order to identify each individual character in a document.

In the early days of OCR, this technology was quite primitive and required the use of a special font set in order to work, but modern OCR technology is no longer limited to this and is even capable of recognizing handwriting in addition to digital font sets.

What are the Challenges of Converting PDF to Text?

When a document is scanned or otherwise converted into a PDF, there are numerous challenges that must be overcome to turn the original document into a searchable PDF. One of the most immediate issues is that there is no universally consistent scanned document - a book, a legal document, a poster, and images of text may contain writing in many different forms, shapes, and sizes. 

 must be able to recognize text however it is present on a page.

 or a low resolution and it may come in virtually any language. Being able to recognize text means being able to recognize not only the Latin alphabet but also the many other writing systems in use around the world. One of the challenges of OCR that must be solved is to be able to recognize all of these characters, shapes, and symbols in varying degrees of fidelity.

Once a document has been scanned into a program such as Adobe Acrobat, the OCR software must perform a number of functions in order to successfully turn a document into searchable text. One challenge that must be overcome is to identify text when it has been rotated - for example, if care has not been taken, scanned documents can often end up being scanned at an angle. The method for dealing with this is to simply digitally rotate the document until the characters match with known patterns. Since documents are usually (but not always) written using a consistent horizontal baseline, once the angle of the characters is known, that rotation can be accounted for and text can be recognized as if it were on a straight line.

The challenge of identifying characters in a variety of writing systems used to be slightly more complex when OCR relied on using specific font sets, but with the advent of Unicode, it too has become rather trivial. The goal of Unicode is to represent and provide for the encoding of as many characters of as many writing systems as possible. This has transformed foreign language recognition into exactly the same process of character pattern recognition as the language locale of your particular device. Whether the document is written using Latin, Cyrillic, or Hangeul characters, the same basic principles apply.

Text Recognition for PDFs: How Does It Work? 

In recent years, social media sites have become widespread and the number of people using them has been large and it continues to grow. People now prefer news pages on social media sites to follow the news, and those pages have become the only news source for a large number of people. As a result, it has become easy to spread fake news and rumors and maliciously mislead people. To avoid those dangers, systems that automatically detect fake news are needed.

 which is published by Delhi Technological University. After that, we will extract more data about the tweets using Twitter API. Subsequently, we will use the enriched dataset for various visualizations to get further insights into the data. Finally, we will train 2 different supervised learning models to distinguish fake news automatically. One is machine learning-based and the other is deep learning-based. 

ECTF is a dataset for Twitter fake news detection in the Covid-19 domain. It consists of 3 files; fake.csv which contains fake tweets, genuine.csv which contains real tweets, and unlabelled.csv which contains mixed unlabeled data. In this tutorial, we will use both 

. Each file of them contains 2,000 tweets with the following fields:

: Each object within Twitter - a Tweet, Direct Message, User, List, and so on - has a unique ID. Twitter IDs are unique 64-bit unsigned integers, which are based on time. We will use this field to retrieve more data about the tweets.

The Twitter API is a set of programmatic endpoints that can be used to access the data on Twitter programmatically, which is precious for data-centric research. This API allows us to find and retrieve different resources on Twitter, like tweets, users, trends,.... etc. On the other hand, we have Tweepy, which is a Python library for accessing the Twitter API.

Log in to your Twitter account and go to the 

. Then, create a new project, and go to the 

. Click the buttons shown in the following image to generate the tokens and save them in a safe place:

All the codes in this tutorial are tested against the following versions of packages:

First of all, let us start by creating a new Conda environment named fnd and activating it:

conda create -n fnd python=3.7
conda activate fnd

Now, we can install the required packages for the first part of the tutorial:

# in fnd environment
conda install -c conda-forge tweepy
conda install -c anaconda pandas

Firstly, let us download the dataset from 

 and then convert them to Pandas data frame using the following code snippet:

import pandas as pd

real_data = pd.read_csv('genuine.csv')
fake_data = pd.read_csv('fake.csv')

To be able to use Twitter API we have to initialize a Client object using the tokens we got in the previous section:

import tweepy
import requests

CONSUMER_API_KEY = "PUT_YOUR_CONSUMER_API_KEY"
CONSUMER_API_KEY_SECRET = "PUT_YOUR_CONSUMER_API_KEY_SECRET"

ACCESS_TOKEN = "PUT_YOUR_ACCESS_TOKEN"
ACCESS_TOKEN_SECRET = "PUT_YOUR_ACCESS_TOKEN_SECRET"

BEARER = "PUT_YOUR_BEARER"

client = tweepy.Client(bearer_token=BEARER, consumer_key=CONSUMER_API_KEY,
                      consumer_secret=CONSUMER_API_KEY_SECRET,
                      access_token=ACCESS_TOKEN,
                      access_token_secret=ACCESS_TOKEN_SECRET,
                      return_type=requests.Response
                      )

Now, since we have access to resources on Twitter, we can start fetching the data for our analysis. To get more information about the tweets depending on tweet_id we can use the 

 function. Let's start by defining some constants;

batch_size = 100
tweet_fields = ["public_metrics", "created_at", "entities", "author_id"]
user_fields = ["name", "username", "location", "public_metrics"]

To fetch the attributes for the whole dataset we have to split it into batches with a batch size of 100 and request the data batch by batch. We can achieve that using the following functions:

def batch(iterable, n=1):
   l = len(iterable)
   for ndx in range(0, l, n):
       yield iterable[ndx:min(ndx + n, l)]
-	Function to retrieve data using get_tweets() function:
def fetch_tweet_fields(dataset):
   all_tweets = dict((k, []) for k in ["data", "users", "errors"])
  
   ids = list(dataset["id"])   
   for id_batch in batch(ids, batch_size):
       tweet_batch = client.get_tweets(id_batch,
                       tweet_fields= tweet_fields,
                       user_fields=user_fields,
                       expansions="author_id",
                       user_auth=True
                       )

       if tweet_batch.json().get("data"): all_tweets["data"].extend(tweet_batch.json()["data"])
       if tweet_batch.json().get("includes"): all_tweets["users"].extend(tweet_batch.json()["includes"]["users"])
       if tweet_batch.json().get("errors"): all_tweets["errors"].extend(tweet_batch.json()["errors"])

      
   return all_tweets

We can call the functions in the following way:

real_tweets = fetch_tweet_fields(real_data)

fake_tweets = fetch_tweet_fields(fake_data)

are dicts that have 3 keys; "data", "users" and "errors". 

list of json objects that contain details about the tweets

list of json objects that contain details about the authors

list of json objects that contain details about the error messages

Note: The errors field shows us that some of the tweets have been deleted or made private. In this situation, we can only retrieve data of 1913 tweets from the real ones and 1030 tweets from the fake ones. This can be an important meaningful indication that lots of fake news are deleted.

For easier operations, we can convert the dictionaries to pandas data frames in the following way:

real_tweets_data = pd.json_normalize(real_tweets['data'])
real_tweets_users = pd.json_normalize(real_tweets['users'])

fake_tweets_data = pd.json_normalize(fake_tweets['data'])
fake_tweets_users = pd.json_normalize(fake_tweets['users'])

Step 5 - Data Pre-Processing and Cleaning

To avoid confusion, we will rename the id field in data dataframe to tweet_id and the id field of users dataframe to author_id:

real_tweets_data.rename(columns={"id":"tweet_id"}, inplace=True)
fake_tweets_data.rename(columns={"id":"tweet_id"}, inplace=True)

real_tweets_users.rename(columns={"id":"author_id"}, inplace=True)
fake_tweets_users.rename(columns={"id":"author_id"}, inplace=True)

Since we have repeated user names, we can remove duplicated ones using drop_duplicates() function:

real_tweets_users =
real_tweets_users.loc[real_tweets_users.astype(str).drop_duplicates(subset='author_id', keep="last").index]
fake_tweets_users = 
fake_tweets_users.loc[fake_tweets_users.astype(str).drop_duplicates(subset='author_id', keep="last").index]

For easier processing, we can merge data and users' dataframes on the author_id field.

merged_real_tweets=pd.merge(real_tweets_data, real_tweets_users, on='author_id')
merged_fake_tweets=pd.merge(fake_tweets_data, fake_tweets_users, on='author_id')

Since we got the data from Twitter API, some of the names are long and not practical to use, we can beautify names using the following script:

def beautify_field_names(data):
   data = data.rename(columns={"entities.hashtags":"hashtags",
                       "entities.mentions":"mentions",
                       "entities.urls":"urls",
                       "public_metrics.retweet_count": "retweet_count",
                       "public_metrics.reply_count": "reply_count",
                       "public_metrics.like_count": "like_count",
                       "public_metrics.quote_count": "quote_count",
                       "entities.annotations": "annotations",
                       "name":"retweeter_name",
                       "username":"retweeter_username",
                       "public_metrics.followers_count":"retweeter_followers_count",
                       "public_metrics.following_count":"retweeter_following_count",
                       "public_metrics.tweet_count":"retweeter_tweet_count",
                       "public_metrics.listed_count":"retweeter_listed_count"}, inplace=True)

beautify_field_names(merged_real_tweets)
beautify_field_names(merged_fake_tweets)

Since we got the data from Twitter API, fields that contain lists are downloaded as json with a lot of details that are not important in the context of this tutorial. Also, some of the fields are not available for all records. For example, location info is available only for some of the tweets and the others are None. It is better to replace Nones with empty strings to avoid string operation errors. So, we can convert lists of json objects to lists and replace Nones with empty strings:

def beautify_items_with_lists(data):
   data["location"] = data["location"].fillna("")
   data["hashtags"] = data["hashtags"].fillna("").apply(list)
   data["mentions"] = data["mentions"].fillna("").apply(list)
   data["urls"] = data["urls"].fillna("").apply(list)
   data["annotations"] = data["annotations"].fillna("").apply(list)
   data["hashtags"] = data["hashtags"].apply(lambda x: [item["tag"] for item in x])
   data["mentions"] = data["mentions"].apply(lambda x: [item["username"] for item in x])
   data["urls"] = data["urls"].apply(lambda x: [item["url"] for item in x])

beautify_items_with_lists(merged_real_tweets)
beautify_items_with_lists(merged_fake_tweets)

A lot of tweets contain links in their texts which can affect the text-based analysis that we are going to do. Since we have URLs in a separate column, we can remove them from the texts without data loss.

merged_real_tweets["text"] = merged_real_tweets['text'].replace(r'http\S+', '', regex=True).replace(r'www\S+', '', regex=True)
merged_fake_tweets["text"] = merged_fake_tweets['text'].replace(r'http\S+', '', regex=True).replace(r'www\S+', '', regex=True)

A big part of the dataset contains retweeted tweets and it will be more meaningful to analyze the original tweeters instead of the retweeters. Since the retweeted tweets start with RT @username, we can retrieve the original user name depending on the tweet text without using Twitter API:

def get_owner_username(tweet):
   text = tweet["text"]
   if text.startswith("RT @"):
       return text[text.index("@")+1:text.index(":")]
   else:
       return tweet["retweeter_username"]

merged_real_tweets['owner_username'] = merged_real_tweets.apply(lambda item: get_owner_username(item), axis=1)
merged_fake_tweets['owner_username'] = merged_fake_tweets.apply(lambda item: get_owner_username(item), axis=1)

Since we have too many fields in the data frame, we are going to select a subset of features to analyze in this tutorial.

sub_set = ['text', 'mentions', 'urls', 'hashtags', 'retweet_count', 'reply_count', 'like_count',
          'owner_username', 'retweeter_username', 'location', 'retweeter_followers_count',
          'retweeter_following_count', 'retweeter_tweet_count']
sub_real_tweets = merged_real_tweets[sub_set]
sub_fake_tweets = merged_fake_tweets[sub_set]

 is a Python library for processing textual data. It provides a simple API for diving into common natural language processing (NLP) tasks such as part-of-speech tagging, noun phrase extraction, sentiment analysis, classification, translation, and more.

We will use this package to extract semantic features from the dataset, like objectivity and polarity scores depending on the texts of the tweets. Let's start by installing the textblob conda package.

# in fnd environment
conda install -c conda-forge textblob -y

To get subjectivity scores we can use sentiment.subjectivity attribute, which returns afloat within the range [0.0, 1.0] where 0.0 is very objective and 1.0 is very subjective.

from textblob import TextBlob

def getSubjectivity(text):
  return TextBlob(text).sentiment.subjectivity

In a similar way, we can extract polarity scores by using the sentiment.polarity attribute, which returns afloat within the range [-1.0, 1.0] where -1 defines a negative sentiment and 1 defines a positive sentiment.

def getPolarity(text):
   return TextBlob(text).sentiment.polarity

Now, we can calculate subjectivity and polarity scores and add them to the data frames:

real_polarities = [getPolarity(content["text"]) for index, content in sub_real_tweets.iterrows()]
real_subjectivities = [getSubjectivity(content["text"]) for index, content in sub_real_tweets.iterrows()]
sub_real_tweets["polarity"] = real_polarities
sub_real_tweets["subjectivity"] = real_subjectivities

fake_polarities = [getPolarity(content["text"]) for index, content in sub_fake_tweets.iterrows()]
fake_subjectivities = [getSubjectivity(content["text"]) for index, content in sub_fake_tweets.iterrows()]
sub_fake_tweets["polarity"] = fake_polarities
sub_fake_tweets["subjectivity"] = fake_subjectivities

polarity_labels = ["Positive", "Negative", "Neutral"]
subjectivity_labels = ["Subjective", "Objective", "Neutral"]

Subsequently, we can label records depending on polarity and subjectivity scores:

pos_size_real = sub_real_tweets.loc[sub_real_tweets['polarity']>0].shape[0]
neg_size_real = sub_real_tweets.loc[sub_real_tweets['polarity']<0].shape[0]
neu_size_real = sub_real_tweets.loc[sub_real_tweets['polarity']==0].shape[0]
sizes_real = [pos_size_real, neg_size_real, neu_size_real]

pos_size_fake = sub_fake_tweets.loc[sub_fake_tweets['polarity']>0].shape[0]
neg_size_fake = sub_fake_tweets.loc[sub_fake_tweets['polarity']<0].shape[0]
neu_size_fake = sub_fake_tweets.loc[sub_fake_tweets['polarity']==0].shape[0]
sizes_fake = [pos_size_fake, neg_size_fake, neu_size_fake]

Now, we can plot the number of records per class as a pie chart:

import matplotlib.pyplot as plt
fig1, ax1 = plt.subplots(nrows=1, ncols=2, figsize=(12,7))

ax1[0].pie(sizes_real,  labels=polarity_labels, autopct='%1.1f%%',
       shadow=True, startangle=90)

ax1[0].axis('equal')
ax1[0].title.set_text("Real News")

ax1[1].pie(sizes_fake,  labels=polarity_labels, autopct='%1.1f%%',
       shadow=True, startangle=90)
ax1[1].axis('equal')
ax1[1].title.set_text("Fake News")

plt.show()

Obviously, we can say that real news tends to be more positive than fake news. On the other hand, fake news tends to be more neutral than fake news.

If we do the same analysis on the subjectivity field, we get the following results:

Conversely, we cannot draw similar conclusions from subjectivity analysis since the results are very close.

We help customers to build AI that matters across industries

They trust us for their data-centric journey

Get Started