Ready to see how it works? Begin your AI journey at the click of button.

Get started

How Eidos-Montréal uses machine learning to leverage the Voice of the Customer and make better strategic decisions

How Eidos-Montréal uses machine learning to leverage the Voice of the Customer and make better strategic decisions Eidos-Montréal is a Canadian video game developer based in Montréal, Canada. The studio, created in 2007 and counting 400+ employees today, is responsible for developing some of the most popular video games of all time including the Deus Ex series, the recent Marvel’s Guardians of the Galaxy, Shadow of the Tomb Raider, and many more.

The Eidos-Montréal Analytics and Data Science department were created in 2018 with a set of clear and ambitious objectives: to evangelize the value of analytics services and the use of data in making business and design decisions. Today, the team provides a source of invaluable insights into what customers and fans think about the games developed by the studio, helping drive the Eidos-Montréal production support department’s strategy and development for their current and future games.

Bringing machine learning to the heart of the business

When Marie de Léséleuc, the Data Science and Analytics Director at Eidos-Montréal, created the department of the same name, her objective was set: to bring data and analytics into the conversation and help Eidos-Montréal make more informed decisions both in a game design and business level.

To do so, they decided to collect customer and journalistic reviews and articles, with the goal to understand what is being said about their games and competitors and turn this information into actionable insights. It was decided that a Machine Learning system would be the right tool to do it.

The Data Science team at Eidos-Montréal started collecting thousands of reviews and articles from Steam, Metacritic, and many other specialized websites. Then, they defined a protocol to classify these reviews and articles. The first data point is the articles and review’s topic (what specific topic is being commented about a game) and the second data point is the sentiment associated with it. The team broke down topics into 18 categories including aesthetic, AI, animation, audio, combats, puzzles, performance, playtime, etc.

For sentiment analysis, they defined three types: positive, neutral, and negative. Then, after a long and thorough analysis of thousands of reviews and articles, they created a list of keywords around these categories. These keywords are used to classify any sentence from a review and, or article, determine which category the sentence belongs to, and identify the sentiment associated with it.

The team leveraged a pre-trained model that knows about general NLP categorization and sentiment as a base for their Machine Learning project. Of course, they had some work to do to fine-tune the model to fit the gaming industry and the specific terms used by their customers to talk about games. Still, it is interesting to note that they did not have to start from scratch. Indeed, more and more models are readily available online, a trend in the ML world. Once their dataset and model were ready, the next phase of the project consisted in training the model to get accurate and consistent results.

Training models on a large volume of high quality, unstructured data 

To make their Machine Learning model accurate, it needed to be trained on thousands of reviews and articles. Furthermore, text, especially the customer’s voice, is complex data to analyze. The model needs to recognize the subtle distinction between negative words that mean a positive sentiment, identify sarcasm, and many other forms of language that are specific to the gaming industry, countries, and cultures.

At first, the team chose to train their model themselves. They used Doccano, an open-source annotation tool for machine learning practitioners. It enabled them to create the labeled data they needed to start training the model. To do the annotations, they relied on the willingness of everyone at Eidos-Montréal.

Also, relying on internal resources to annotate hundreds, if not, thousands of reviews and articles were just not scalable, and the data science team had to spend a lot of time reviewing the quality of the annotations.

The challenges of relying on open-source tools and in-house labeling:

This is why they started looking for a more professional approach. After looking at various providers, they decided to run a POC with Kili to test the solution on their use case.

“It appeared quickly that Kili would be able to answer all our needs for our use case. On top of this, the availability of the customer support team made Kili’s integration into our ML workflow painless and quick.” -Marie de Léséleuc, Data Science and Analytics Director at Eidos-Montréal 

High-quality training data to provide actionable insights

For Eidos-Montréal, the key to the success of their project relied on the quality of the annotations at scale, so their model could provide accurate insights to the different business departments.

Boosting efficiency and scalability with external workforce and quality control.

To help with this, the data science team decided to use a professional workforce provided by Kili as part of Kili’s service agreement and partnerships.

“The labelers that Kili provided to support our project are very good. They are familiar with the jargon of the gaming industry, and they understand very well what we expect of them. Also, Kili’s support team made sure to train them on the platform. This accelerated considerably our progress and improved the quality of the annotations.” - Marie de Léséleuc, Data Science and Analytics Director at Eidos-Montréal 

The other key factors for the team were the tools dedicated to quality control that Kili offered on its platform. As Nika explains it, one of the most important features for them was the consensus feature. By leveraging consensus, they make sure to get consistent labels across all their training datasets. It also helps the labelers to improve their skills much faster.

“The consensus feature on Kili’s platform is exactly what we needed. It made a tremendous amount of difference to control the quality of the labelers’ work and get to a level of quality and consistency at a scale that we could not achieve before.” - Nika Abedi, Data Scientist at Eidos-Montréal 

Autonomy and support accelerate project delivery

The data science team at Eidos-Montréal is small. Being autonomous and dedicating their time to value-adding activities is very important to them. By leveraging Kili’s API, they easily add new data to the platform to add them to the labeling queue or create new projects.

The team also widely credits the easy and intuitive Kili interface to create and follow up on projects, without needing any help. The dashboard provides all the information. They need to know; how many annotations were completed, what’s left in the queue, and what is ready for review.

“Kili’s documentation is also very helpful. On various occasions, it helped me quickly solve issues I encountered without having to contact the support team.” - Nika Abedi, Data Scientist at Eidos-Montréal 

“Our understanding of how to ship highly efficient ML models is maturing and Kili plays a great role in this process. With Kili, we are in control of our datasets and the quality of the training we give to our models. The platform also enables us to quickly and easily iterate and ship new projects.” - Marie de Léséleuc, Data Science and Analytics Director at Eidos-Montréal 

Empowering business and devs decision-makers with reliable data

The outcome of this investment is a source of invaluable insights coming straight from their customers, that empowers the teams across the business and development departments to make strategic and operational decisions. The data-science team built dashboards that filter the data and provide analysis of what customers think regarding many aspects of their strategy and products. Recent insights gathered by the tool were compared with social media research collected by the performance marketing team at Eidos-Montréal, which showed a consensus on the findings - enabling the teams to confidently present the intelligence to leadership.

“This initiative is a real success for us. We deliver valuable insights about market trends to our strategy teams. We help marketers better understand what our customers and the market like or dislike about our products and the competition. We support our developers by providing them with direct feedback from our customer base about very specific features in our games.” - Marie de Léséleuc, Data Science and Analytics Director at Eidos Montreal 

With such rapid success, the team is rightfully eager to evolve its capabilities. Ultimately, the team’s roadmap is to keep scaling by always analyzing more reviews and articles, adding more granularity, and opening new sources such as more countries and languages. Then, their goal is to build an automated, continuous improvement floe to train and run their models, all in the cloud for maximum agility, from scraping the data to running the model and providing actionable and valuable insights to the teams.

Gaming - Voice of the Customer

Eidos

Get Started

A global manufacturing company was exploring ways to reduce the human effort of manual inspection of defaulted products due to the high cost and lead time, causing bottlenecks and delays in the assembly lines. The company looked into automated defect detection with artificial intelligence (AI), and Kili Technology was involved in training the AI to detect defective products through data annotation and machine learning loop in the process.

Detecting thin cracks and scratches with computer vision

As a global manufacturing company headquartered in France, quality assurance on the assembly line is one of the utmost importance to ensure perfection within the promised lead time delivery for customer satisfaction worldwide. However, as the company was doing manual quality checks with human inspectors one by one per unit product, the process became heavily time-consuming, costly, and required huge efforts that could take hours. This resulted in production bottlenecks and increased lead time. On top of that, manual inspection contained the risk of missing minor defects in the component unit, where each unit is small hence scratches or broken parts are sometimes undetected by human eyes.

“Defaulted components often went unnoticed and got assembled with other components, and became defective products. What’s next: customer complaints, returns, and replacements, costly consequences.”

Using artificial intelligence as the core of its diagnostic tool, the company understood perfectly that the accuracy of data to train the model is highly critical. It was already clear that to be able to provide a precise diagnosis of bladder cancer, the AI model needs to be trained with clean, accurately labeled images of cytology slides. Thus, the image labeling process is essential as it would determine the precision level of the diagnosis, which will impact people’s lives.

Artificial intelligence is considered to be one feasible solution to significantly improve the process and therefore reduce the impact of defective products on customers by both preventive and reactive measures. Using a computer vision model, defect detection on the component level would prevent the defaulted unit to be assembled to become a defective product. On top of that, defective products are also detected to be taken away directly with robotic arms from being packaged. To implement this solution, the company needed to ensure that the AI model is trained well to deliver an accurate detection of defects.

The challenges of building an AI defect detection

To build the AI model that can accurately detect defective products in assembly lines as the company envisioned, it found several challenges, which mostly involved the AI model training process. One challenge was to build excellent quality training datasets that consist of cleanly labeled images and videos of well-conditioned and defective products so that the AI model could identify defects and act properly.

Another challenge is to significantly increase the speed and efficiency of the training process of the AI model. As the manufacturing company adopted lean production as its manufacturing philosophy, the company struggled to find ways to apply this value to be more efficient on AI model training – to achieve the same level of accuracy with less training data.

“As we are used to being lean, there should be a unique 80/20 method to be efficient while still being rigorous in the quality when training the AI model (for defect detection). We were unsure what is the effective way to do this” – 

Mélanie, Head of Digital Transformation.

As the challenges focus on being lean to achieve an excellent AI model performance at scale, the manufacturing company understood that it needed to find a partner that could offer simplicity and efficiency while delivering a high quality of training data. The company first explored Kili through the free trial offer and tested the tool.

“We tested the tool rigorously initially on our own. Then, after the introduction, Kili set up a dedicated testing project for us and even shipped features specific to our needs. Certainly, I was impressed.” – 

– Mélanie, Head of Digital Transformation.

Deciding to partner with Kili afterward, the company found that the simplicity of the tool helped to be productive in quickly building training datasets for its AI model for defect detection. The customizable interface and features addressed directly to the company need to make it intuitive to navigate, improving efficiency.

Furthermore, the company was excited to see that machine learning in the loop – notably active learning when annotating data on the Kili platform – became the efficient 80/20 method the company needed to train the AI model with 30% fewer data to achieve high accuracy of performance. In addition, Kili facilitated the company to leverage online learning to create automated predictions in the labeling process, doubling up the data training speed.

Having implemented the AI automated defect detection systems in its assembly lines, the manufacturing company realized the positive impact on cost-saving and increased employee productivity. The AI model also pulled up the rate of defects found in components by 50% compared to manual inspection by humans, preventing them from being assembled to become faulty products.

The newly enabled product defect prevention also facilitated the company to significantly reduce the number of products returned by customers, demand for reimbursement, or product replacement.

"We have prevented faulty components to be assembled and defective products to be shipped – which allows us to reduce cost by 25%. I certainly see the impact, positive results are here.” 

Automating defect detection with AI in manufacturing company

VitaDX, a French healthcare company focusing on the field of bladder cancer diagnosis is developing a new AI-powered diagnostics model to provide a simple, reliable, non-invasive, and more efficient diagnostics tool to detect early bladder cancer as a complementary for urinary cytology. However, the company found it challenging to industrialize its data pre-processing to build the tool while managing the data accuracy in check – as accuracy is highly critical in medical diagnostics. VitaDX partnered with Kili Technology to enable not only a simplified process but also a more structured, collaborative workflow.

Aiming for perfection: when accurate diagnosis could save people’s lives

Globally, almost 3 million people are currently diagnosed or treated for bladder cancer, with 400 thousand global new cases each year. Like almost all other cancer types, the survival rate of a bladder cancer patient increases when it is detected earlier. However, urinary cytology – the non-invasive diagnostic method available – has a poor performance in detecting early bladder cancer. As a company, VitaDX has a mission to build a new tool to significantly increase the capability to accurately diagnose early bladder cancer with a non-invasive method, using an image processing algorithm of urinary cytology slides, scanned in white light and fluorescence.

“We build an artificial intelligence model that will determine whether a person is sick. At the end of the day, our model will impact the life of a human being. This is why it needs to be accurate, quality should be the first to prioritize since the beginning – data preparation.” 

The challenge of diagnosing cytology slides at a scale

Before partnering with Kili, the company relied on crowdsourcing to annotate cytology slides. On the face value, the cost of crowdsourcing seemed lower as all of the labelers worked part-time. The annotation process was also done without a dedicated resource in charge of data operations. Thus, each crowdsourced labeler had his annotation stream and flow. However, VitaDX realized that when it comes to delivery, it was extra difficult to achieve the level of annotation quality required and also to manage the consistency of this quality. With no one in charge of data operations, monitoring these crowdsourced labelers was a challenging task.

Another challenge is that VitaDX understood that the medical image annotation process is laborious, and highly time-consuming work. Annotating cytology slides involves performing semantic segmentation of the images, and this requires pixel-level precision. Then, each labeler needs to label carefully with laser-sharp focus, resulting in a long process to label one image. Reviewing the labeled cytology slides was also tiresome, as doctors need to sit together and discuss when there is a disagreement over a label of presumably cancer cells.

“Precision is very important, it determines the quality. The quality of annotation will impact on the quality of our machine learning model, and at the end the quality of the medical device where we are building.” continued Abderrahmane.

Reflecting on these challenges, VitaDX realized that to perform cancer diagnosis at scale it needs to find a solution to structure and streamline the process of annotating cytology slides and the collaboration between doctors, machine learning team, and data labelers. Benchmarking against several top players in the market, the company found Kili offers a combination of features suitable to their needs. More importantly, Kili stood out when it comes to quality management of the annotations.

“We found the quality management features – especially consensus – very useful, as it allows us (doctors) to gather up only when an asset has a low level of consensus, which means the cell image is very complex to identify. Otherwise, there is no need to review together, hence it saves us time.” – 

Partnering with Kili, VitaDX was able to structure the annotation process much more clearly. One person was required to be pointed as the person in charge of data operations for quality management feature on Kili, hence quality control is kept in check. Reviewing features on Kili also allows a more efficient collaboration for doctors, machine learning teams, and data labelers, breaking the SILOs of departments that existed previously.

Moreover, the company found itself to be able to perform image labeling much more precisely at a way faster pace. Using online learning as a machine learning in the loop was enabled during the process, where VitaDX plugged in its machine learning model into Kili to automate the labeling process itself. This process allowed a pre-annotation process, where labeling predictions were produced, reducing the effort of the annotators. Kili was also flexible to use in devices suitable to deliver pixel-precise perfection quality.

“We speed up the process five times.”

Industrializing image annotation of cytology slides to detect bladder cancer enabled VitaDX to perform cancer diagnosis early, more accurately, and at scale. Streamlining the whole labeling process along with online learning as machine learning in the loop reduces the time taken to annotate images by 70%, and it improves the efficiency and productivity of the team.

The walls between SILOs of departments that existed previously, hindering collaboration and transparency of workflow, are also gone – drastically improving productivity and accelerating the diagnosis process. The overall cost to conduct cancer diagnosis is reduced.

“If we don’t have high-quality data, the process of our diagnostic device could have been much slower. The bonus is that we speed up the process five times” 

Enabling innovative accurate diagnostics for early bladder cancer detection

A European healthcare technology company focusing on innovations for ophthalmology aims to make eye screening more accessible to a wider public with accelerated speed and reduced cost. The goal is to improve care for eye patients and to detect eye disease as early as possible. To proceed at scale, AI is adopted at the core of the eye screening model, and the company partnered with 

to ensure the accuracy of this model with clean labeled retinal data of patients.

The mission is to improve people’s vision at a scale

Today, millions of people in the world are at risk or already have vision-threatening eye conditions that are left undetected, such as diabetic retinopathy or glaucoma. Some people even have eye conditions that are rooted in even more serious life-threatening causes such as stroke or cardiovascular disease – which also are left undetected. While traditional eye screenings are fairly costly and time-consuming, with artificial intelligence the company has the ambition to accelerate this process to provide access to the wide public and detect all possible eye conditions within seconds at low cost, at scale.

To deliver precise eye diagnosis within seconds, the company acknowledged that its AI eye screening model needed to be trained rigorously and enriched with a combination of accurately annotated data – such as retinal photography, CT scan images, and medical reports. Thus, the data annotation process is essential to perform eye screening at scale to achieve the company's mission to improve people’s vision.

The challenge to enriching the AI eye screening model

Understanding the variety of patient data needed to diagnose eye conditions, the company realized the complexity and challenge to train the AI eye screening model. One of the challenges is finding a solution that is fully compatible with the different data types, especially for specific ones such as CT scan data. The company found it difficult to manage when they needed to host the labeling process with different tools to annotate different data, as each tool has a different flow some are more complex than the other, and the whole process became inefficient.

I don’t think many tools can annotate our data types. There were not many that can perform up to our standards as well. Let me tell you, using different tools at once can get your project to be all over the place. It was a headache.

Another challenge is the need to speed up the labeling process, as to enrich the AI model, they need to only train rigorously in terms of rounded data types, but also terms of voluminous amount – while ensuring precise quality. When the team annotates retinal and CT scan images, these images need to be cut down into manageable sizes to prevent being lost and missing details. Therefore, the total amount of training datasets multiplied significantly, and the annotation process and ensuring quality is certainly time-consuming.

To solve these challenges, the healthcare company benchmarked Kili against multiple players in the market, including the different ones the company adopted previously. The company considered it highly practical when it found that Kili could accommodate all the data types needed – including CT scans format. On top of that, the customizable interface and reviewing features also improved their collaboration and flow of data annotation work.

as very efficient in being fast while keeping a high standard of quality of annotation. The ability to plug in machine learning in the loop within the Kili platform for the annotation process speeds up the annotation significantly. Also, the quality management features such as consensus and honeypot facilitated the company to scrutinize the quality of data labels effectively.

“It’s very convenient and simple to use. And when it’s simple to use, you get to be extremely more productive. No more headaches.” continues Alexander in his remarks on using Kili.

“Our mission is achieved – and will continue”

Simplifying and scaling up the data annotation process has enabled the healthcare technology company to perform fast delivery of eye diagnosis and to accurately detect potentially dangerous eye conditions early – at scale. Previously, patients who undergo eye diagnosis would receive the results in days or even weeks. With an artificial intelligence model enriched with labeled training data, now patients the result can be obtained in real-time within a minute. It also reduces the cost of performing eye diagnosis by 40%, making it more accessible to the wider public.

It’s exciting to see our model perform at hospitals and eye clinics. People get diagnosed within a minute. Our mission is achieved and certainly will continue.

Accelerating retinal disease detection at low cost

A large European banking client found itself having limited capability when handling customer service functions manually by answering phone calls. Customers need to wait in lines and wait for long periods which lowers consumer satisfaction and loyalty. When the bank decided to implement artificial intelligence to build chatbots at scale, 

became its trusted partner to facilitate training datasets to enrich the quality of the chatbot and monitoring in production.

The need for chatbots to accompany customers

Like most banking institutions at the time, a large European bank client found itself needing to handle customer service to answer daily questions or complaints from customers through phone calls by customer service operators. The old usual process was typically illustrated by a customer dialing into a customer service number and having to wait in line until the operator picks up – and having to listen to a random “waiting song” repeated several times over. As the volume of customers dialing in is always far greater than the operators handling the calls, to customers this “waiting song” is playing in the loop without end.

“We conducted a survey of our banking customers, and major feedback was given to how long they had to wait just to ask questions. There was a barrier – we were unreachable to them,” explains Emma, the Head of Customer Success at the bank.

Photo credits to katemangostar – fr.freepik.com

When these customers have urgent queries, this unreachability became a pivotal point against the bank, as they see their customers thinking to turn heads to alternatives. To solve this issue, an AI chatbot development at scale was considered a highly attractive alternative to the old-fashioned customer service calls. This AI chatbot would erase the high barrier between the bank and customers easily, responding to each one of the customer’s questions and messages in real-time, at all times. Additionally, the chatbot also allows self-care development, in which customers can get their answers quickly themselves and only escalate the conversation to be delegated to human operators only when it is added value.

The challenges of building an AI chatbot at scale

Building an AI chatbot is not without its challenges. First, it is important to have a comprehensive list of intent that is clearly well-defined and distinctive in each one to be recognizable by the chatbot. The challenge in this list of intent then comes to creating the variation of intent from the original list, which covers a wide range and is exhaustive enough for the AI model to become robust.

Moreover, another challenge is to facilitate a supervision function to the conversation conducted by the chatbot, be it cold supervision or live monitoring by a person – with a room for even the supervisor to jump in. This supervision would involve grading the chatbot conversation quality – whether the conversation was relevant – and the quality of the bot, whether the chatbot could handle the conversation properly, and if the bot could escalate the issue by delegating to human customer service.

…the point is to be able to investigate where the AI bot got lost and tell whether this part of the conversation should have been understood well. If we find gaps between intent, we shall add this missing intent so next time the bot will perfectly understand.

To address these challenges, the bank then needed to make sure that the AI chatbot is trained rigorously with bulk amount datasets that are accurately labeled entities covering the variation of intent in multiple languages.

As the bank is already aware, the data annotation process is vital to produce high-quality datasets to train the chatbot to understand the variety of intent. Based on the challenges mapped, the bank was looking for a data annotation company that could provide a simple tool with rigorous quality management and flexibility in collaboration. After testing several tools offered in the market, the bank considered 

Working with Kili, the bank found it useful to leverage advanced features such as consensus and honeypot to control the data quality to make sure utterances are interpreted well with the correct intent. On top of that, the Kili platform also simplifies the collaboration between data scientists and annotators during the labeling process, making learning and iteration quicker.

Using chatbots, the bank sees huge cost savings. The cost with customers, when conducted with a human operator, costs 3 euros/interaction. The bank was conducting 1 million interactions as a basis to produce another 4 million incremental interactions. With human operators, this would cost EUR 3 million. Implementing the chatbot has reduced the need for human interaction by 50%, therefore saving costs by EUR 1.5 million.

The impact is massive. We receive customer compliments praising our better customer service. We got around 60% ROI, which is pretty well. It’s definitely worth it.

 to discover more on how we can create an impact at your company.

Scaling up banking customer service with chatbot

For years, a major French banking institution was having a bottleneck in inefficiency when it comes to handling emails processing for customer service. A fully manual procedure with human personnel was very time-consuming and costly, and the productivity level per employee was expected to increase. When the bank decided to proceed with email processing powered by artificial intelligence (AI), it partnered with 

to train the automation models with high-quality datasets.

Realizing the importance of artificial intelligence automation

As one of the leading European banking institutions since the 80s, the bank serves millions of customers worldwide. Every day, the bank receives thousands of emails from customers, be it in form of customer complaints, questions or inquiries over banking products, registrations, document attachments follow up, among others. Regarding this, the bank was facing one problem that led to a significant bottleneck in inefficiency: the time and effort taken to process those emails manually.

“It takes hours for my staff to read the emails and check the attachments, only to forward it to another staff as the email he opened is not relevant to him and so he couldn’t respond directly.” Explained Guillaume, Director of Technology and Innovation at the bank.

This time-consuming process prevented the employees to allocate time to more important tasks, which resulted in low productivity. As margins are under pressure, it became a problematic issue for the bank to be solved. At the time, AI-powered emailing automation was seen as a solution that could streamline and simplify the process. This AI automation would enable automatic classification and routing of emails to relevant employees, thus saving hours of work. To transform the idea into reality, there are a few challenges to tackle in building this automation model.

The challenges of building the AI automation model

Building an artificial intelligence model for email processing is not without its challenges. First of all, it is challenging to create a classification scheme that is simple enough to be understood by the AI model. Second, there should be a strategy that allows the bank to locate risk on mistakes in classification.

When two or more staff read the same email, they might have different agreements on who they should route the email to. This represents the risk of a mistake that the AI model can make. Hence we need to foresee this to make the model accurate and efficient.

Moreover, another challenge is to facilitate the AI automation model to have room for data correction and reclassification of data when there is a mistake in identifying information or attached documents within the email. Human feedback can enrich the model and make it more robust. To address these challenges, the bank then needed to make sure that the AI model is trained rigorously with datasets that are accurately labeled to ensure the high-quality output of performance.

Kicking off with the project, the bank formed partnerships with companies. Aside from choosing AI consultancy and implementation service, one key strategic partner is a data annotation tool and service. As the bank is already aware, the data annotation process is vital to produce high-quality datasets to train the AI model to be able to avoid mistakes during work. Based on the challenges mapped, the bank was looking for a data annotation company that could provide a simple tool with rigorous quality management and flexibility in collaboration.

“We assessed several providers available in the market and tested each of them. 

stood out as the best solution that could best facilitate our needs” explained Guillaume.

Choosing the most suitable data annotation solution mainly revolved around ROI. The bank needed to assess the cost-benefit analysis very carefully to make sure the return is favorable for them when they invest. In addition, the bank also paid attention to data privacy and the security of sensitive documents.

Working with Kili, the bank found it useful to leverage advanced features such as consensus and honeypot to control the data quality, aside from its advanced technical labeling features. On top of that, the Kili platform also simplifies the collaboration between data scientists and annotators during the labeling process, making learning and iteration quicker. Data security is highly protected and guaranteed.

After the project is implemented, currently 92% of the customer emails are processed with automation. The bank’s customer service employees feel a positive impact on their productivity, as more time can be allocated to more urgent tasks than reading emails and opening attachments.

Solving customer inquiries through email previously took 20 minutes, with automated email classification it takes only 7 minutes – including providing answers and solving the issue for the customer. It saved approximately 14,560 hours of employee work per year in branches in France only. This equates to approximately 10 FTEs. Moreover, human in the loop during the model training increases data accuracy from 80% to 85%, which allows for a significant productivity improvement.

It’s exciting to see the impact. It’s the impact that matters. We don’t only scale up our processes to be extremely efficient and reduce our costs, but we also increase the satisfaction of both customers and employees towards the bank.

Increasing productivity of banking customer email processing

Jellysmack is a multinational social video company based in France. Their technology helps video creators drive massive social audience growth and amplify monetization. Their catalog of 300+ creators in various categories from gaming to sports and beauty generates +10B monthly views across all social platforms, including Youtube, Facebook, Snapchat, Instagram, Twitter, and TikTok.

Jellysmack needed more than their in-house solution to train their models and label data at scale

Jellysmack started its ML journey by analyzing video performances on Facebook. They had to evaluate how much they should spend on ads and position their content most profitably. A video that is 10% less effective in retention divides the number of views by a scale of 100; that’s why video quality is critical. They quickly expanded to other platforms, including Youtube, Instagram, Tiktok, and Snapchat. Today, their algorithm analyzes more than 638 million videos continuously (just on Youtube) to help them identify creators with high potential and new trends in viewership. 26 to 30 million videos are added every month to their monitoring catalog.

“Both data and automation are key to our business. We need to quickly identify the creators and the best-performing content and distribute it efficiently. We also need to scale very rapidly.”

Andrea Colonna, Head of Data at Jellysmack 

Jellysmack quickly identified several use cases to get even more insights and strengthen its services. However, they quickly realized that their in-house annotation tools and training model processes would not be sustainable.

”We could have done it internally, but this task is very time-consuming, and we did not have an efficient interface”, explained Andrea. “We decided to partner with Kili to empower our data scientists. Our internal forces quickly became autonomous with their ML projects”.

One central hub to address all their use cases, from NLP to Video annotation

Jellysmack had several use cases to address, including text and video. With Kili, they found a platform to centralize and annotate any type of data, streamlining their training pipeline through one unique training hub.

NLP annotation to automate the content categorization

As a first step, Jellysmack needed a solution to better categorize Youtube videos. Since Youtube does not distinguish categories beyond large verticals like sports or music, Jellysmack needed to differentiate the content with a higher level of granularity (basketball, American football, European football, or tennis instead of “sports”). To build their classification, Jellysmack outsourced the training of their algorithm based on 200-300 Youtube channels (with 10 videos per channel).

Jellysmack partnered with Kili to train the content categorization model using the text description of the videos.

Video annotation for quicker content distribution across platforms

To optimize content distribution and performance, Jellysmack needed its videos available in various formats so they can be published on platforms as different as YouTube, TikTok, or Instagram. To ease their video editors’ work, Kili was requested to train an algorithm to automatically detect transitions and cuts within videos so editors could rapidly identify different content segments and repurpose them.

Today, Jellysmack’s data scientists are using Kili’s video labeling interface to train their model and accurately annotate thousands of videos.

Gaming Beauty Sports Reaction Cooking News …

NLP annotation for sentiment analysis to better understand content performance

Jellysmack wanted to check if a video was performing or not beyond the Like/Dislike ratio. The popularity of a video and the audience’s sentiment is not always reflected by the number of views. That’s why they built an algo in 2019 to understand if comments were positive, neutral, or negative. In the long term, negative comments might impact the performance of a video, even if that specific video has a great number of views. This enables the analytics team to make strategic editorial and business recommendations.

Kili’s platform has become a central hub for all their annotation projects and became the single source for managing all their datasets, which considerably facilitated and accelerated their machine learning workflow.

Jellysmack improved by 5x the performance of their NLP & Video ML models

The collaboration with Kili started in 2018. The reliable support provided by Kili and the smooth integration of the algo developed was certainly the keys to sustainable success.

Kili’s integration into Jellysmack’s workflow was smooth and enabled the team to be operational almost immediately. “Kili’s integration was easy. Their API docs are very well explained, and it enabled us to get operational quickly,” explained Barthelemy Pavy, Data Scientist at Jellysmack.

Another key differentiator for Jellysmack was the level of support provided. “Kili’s customer support is best-in-class and has a direct impact on our performance. It enabled us to launch projects and solve issues much faster than what we were used to.”, added Andrea Colonna, Head of Data at Jellysmack.

Thanks to the smooth integration and great support, Kili became a key contributor to making Jellysmack’s algorithms 4-5x more efficient than it was 4 years ago. This level of enhanced performance enabled Jellysmack to scale with confidence as they expanded their business and helped them provide better services to content creators.

“Kili had a real impact on our roadmap and the level of service we provide to our customers. Kili enabled us to improve our models’ performance and scale our AI projects as fast as our business needs.” - Andrea Colonna, Head of Data at Jellysmack 

Efficient workflows and UX enable data scientists full autonomy, effective collaboration, and scalability

Back in 2018, Jellysmack had about 7 data scientists (DS). In 2021, they reached the milestone of 25+ DS and they expect to get up to 35 by the end of the year. This hyper-growth reflects how important AI is to strengthen Jellysmack’s offering. The growing volume of data processed stands for one of their biggest challenges.

Kili has positioned itself as a key partner in its data science team performance.

“A critical issue at Jellysmack is to scale our capabilities quickly. Kili plays a big part as their solution enables our data scientists to be fully autonomous with building and managing training datasets and ship models into production faster.”

Kili’s intuitive platform and powerful workflows empower DS to manage their projects, increasing their team’s productivity and reducing by a factor of 2 to 10 times the length of the training phase of their models. Data Scientist Barthelemy Pavy mentioned how Kili’s interface made his life easier on a daily basis: “I can easily get an overview of my projects with useful KPIs such as the distribution of my assets and the status of the labeling work, so I can quickly jump onto what will bring the most value to me.”

The solution has been developed at Kili with ongoing improvement, which brings a significant asset to Jellysmack’s Data Scientists team.

“Kili gives us a reliable and powerful video labeling solution that they keep on improving to facilitate our most complex projects. The interface provides all the tools we need to make our workforce more productive and annotate with accuracy.” - 

Thomas Michel, Data Scientist at Jellysmack 

Collaboration is another key factor when annotating complex datasets. To be successful, it is critical that the various project members can work and communicate together efficiently.

“Kili makes it easy to collaborate. We can quickly onboard new project members, and take questions or provide feedback from and to our annotators to quickly solve issues or help them improve their skills.”

Barthelemy Pavy, Data Scientist at Jellysmack 

A collaboration that enabled Jellysmack to address an exponential growth

Addressing localization might be the next challenge for Jellysmack.

 Jellysmack is adding more and more countries to its portfolio, including India and South Korea. Challenges will most certainly arise and will grow beyond the language itself as each culture has its own way of talking and commenting on content. The company will need to train its algorithms to understand these specificities.

This is why Jellysmack follows the data-centric AI revolution with great interest.

 Lots of ready-to-use models do not require much rework internally to fit their use cases. The real challenge lies in the data. To get the expected level of performance, JellySmack will need to gather a significant volume of data and build strong training datasets. Then, without changing the code, continuous work on dataset improvement will be required and altered based on the observed models’ performance. Continuous iteration on the dataset will be key to building better AI, faster.

How Jellysmack improved their NLP and Video models’ performance x5

Reduction in repairing cost due to inspection human error

Reduction in customer follow up update calls

A global insurance company was transforming the procedures of its auto-insurance claim process, especially in the vehicle damage assessment. The goal was to simplify and accelerate the inspection process with equivalent, or if not, better accuracy of repair cost estimation to improve productivity and customer satisfaction. Artificial intelligence was adopted in the digital transformation process of damage assessment, 

 contributed its role in enriching the model with high-quality datasets through data annotation.

When AI speeds up the assessment of car damage and repairment cost

In the old days, the company’s capability to assess damages to estimate the repair cost would involve manual inspection of plenty of not-always-high-resolution photos taken from mobile cameras, phone calls, and onsite physical inspection by assigned assessors. While this process is already time-consuming enough on normal working days where car accidents can happen on busy streets, in holiday seasons where road traffics (and unfortunately, car accidents as well) is peaking – car damage inspection for auto-insurance claim assessment process would take ages. The consequence of this manual inspection process was a scenario where customers had to do repeated calls to check up on progress status – only to hear instructions to call again, with no certainty. It was no surprise that as weeks progressed, the customers’ tone over the call increased with growing impatience.

To prevent turning this impatience into non-subscription, the company considered artificial intelligence as a digital transformation solution to this highly manual damage inspection process. Using a computer vision model, automated detection of damaged body parts can be done very quickly from the severe wrecks to the thin scratches, at scale. More importantly, quick detection of different degrees of damaged car body parts would translate into better visibility and faster calculation of accurate estimation of repairing cost – depending on which part of the car component got wrecked and how bad the damage is.

The challenges of building an AI-powered car damage inspection model

To implement this solution, the company needed to ensure that the AI model is trained well to deliver an accurate detection of defects. The insurance company found several challenges to overcome when building the AI model for car damage assessment. First of all, building a proper dataset itself to train the AI model is a challenge in itself as currently there is no, if not very limited, publicly available dataset of broken or damaged cars. Therefore, the company had to build its very own dataset that needed to be extensive in terms of different types of cars, types of body parts within the car, and also different types of damage along with varying degrees of severity.

It is very difficult to get relevant public datasets of damaged cars that are clearly labeled. Even if we found one, it is not extensive in terms of types of wreckage, or how bad the wreck is. Our AI model needed to be very accurate, then a specifically tailored labeled dataset had to be developed.

More importantly, aside from the quantity of the dataset, the company needed to ensure the quality of these training datasets being developed. Quality inaccuracy contributed a highly critical role as it would determine the repairing cost estimation, and miscalculation of this cost would either produce loss to the company or customer complaints. The company also found it challenging to find a way to increase speed and productivity in training the AI machine, as manual labeling and collaboration between assessors to label the data, data scientists, and the machine learning team would be time-consuming and expensive.

Understanding the challenges it was facing, the company realized the importance of partnership with a company that could offer solutions in training data preparation, especially in terms of data annotation. When exploring numerous data annotation companies available in the market, the company discovered Kili and tried different features of the tool. The company was excited that the end-to-end solution on data annotation at Kili enabled the company to not only annotate the extensive amount of unstructured images of damaged car body parts easily, quickly, and at scale, but it also enabled the company to set up a fluid collaboration between the assessors, data scientists, the AI division team, and the external labelers sourced by Kili as the company needed a handful of experts to help annotate the bulk data.

Moreover, the company favored the application of generating automated predictions of labels by applying online learning in the annotation process. This leveraged the company to speed up the labeling per each image of the damaged car, making the overall development of the training dataset of damaged car images for the AI model to be much faster than the company expected.

It was an end-to-end solution for us: we got not only the very simple tool, but also a powerful API, high-quality experts as external labelers, and automated predictions in the process.

“What once took ages, we made it real-time.”

Adopting artificial intelligence to assess car damages and estimate repair costs as part of the auto insurance claim process, the global insurance company shortened not only the time taken to assess damages, but also the overall end-to-end steps for the customers to process a claim, estimate cost, and get their payment. As a result, company productivity and customer satisfaction skyrocketed. Using an AI model to detect car damages also improved the accuracy of cost estimation, reducing the level of errors done in manual inspection, saving up 17% of the “unforeseen” repairing cost due to inspection errors.

I believe we reduced more than 32% of the number of what we describe as “follow-up calls” where the customers ask for the status update of their claims while damage assessment is ongoing. What once took ages, we made it real-time. We keep on getting positive feedback from customers.

Transforming car damage assessment in auto-insurance claim process

A global insurance company aimed to simplify the steps of processing insurance claim forms, at scale. The objective was to improve the productivity of the insurance claim filing and reduce the complexity of the steps involved in the claiming process. When the company decided to adopt artificial intelligence at the heart of the transformation process, 

 played its part in preparing clean, quality training datasets for AI to build and refine the AI model through data labeling.

The importance to reduce the steps of claim filing with AI

Using the old ways of the insurance claiming process, there had been some negative feedback associated with the long duration, low responsiveness in case of customer interaction, and also lack of transparency of the process. Traditionally, insurance claim forms had to go through at least three steps to be fully processed and approved: submission through an online portal, completion with additional required documents sent manually through the post, and finally review and verification – before the process could continue to the damage assessment process. These steps would normally require days or weeks to be completed, and the highly manual process made it difficult to provide transparency to the process that customers needed.

To transform claim form processing, the company realized it had to simplify and reduce the steps and eliminate the paperwork. The company understood that artificial intelligence models could enable itself to automate the process with natural language processing (NLP) to provide a quick claim filing process in real-time. Using NLP would allow automatic documents classification and customer identification by extracting phrases be it from text or native PDFs, hence eliminating the need for manual document sending and review by human officers. This way, the company would be able to make the claim filing procedure to be much quicker. Additionally, it would also save FTEs costs as using AI would eliminate the need for human officers to manually review the claim filings.

I knew that the steps and the overall process could be simplified, quicker, and simpler. Customers demand speed and transparency, on the status of their documents – but we couldn’t provide it.

The challenges of building AI automation of claim filing

Building an artificial intelligence model to automate a previously highly manual insurance claim filing process was not as simple as the company thought. Firstly, the challenge was to develop a classification scheme and solid strategy so that the NLP model could well-understand and classify the type of documents, client information, and also types of claims in the filings. The mapping of potential misclassifications also had to be done correctly.

Secondly, while the need to train the AI model exhaustively with clean labeled datasets was understood by the insurance company, it was quite difficult to find publicly available datasets that perfectly matched the company's needs, as claim filing usually involved sensitive data. Hence the challenge to build suitable high-quality training datasets for the AI model arose. Furthermore, another challenge was to allow human feedback on data correction and reclassification in the AI model when a mistake in identifying information, document, or type of claim is spotted. A human in the loop was important to enrich and solidify the model.

Moving forward with the development of the AI model, the company understood its need to find an advanced and reliable end-to-end solution for data annotation to build a rich training dataset for the AI model training process, in order to solve the challenges the company was facing. Before acknowledging Kili, the company partnered with another provider, however as complexity progressed the tool couldn’t catch up with its expectations. Moreover, the insurance company struggled to use its initial tool when the human in the loop was involved, as it was not very suitable. Stopping the partnership, the company experimented with annotation tools and decided to pursue the project with Kili Technology.

The company found Kili’s interfaces to be easy to use and customizable to its needs, with an intuitive UX that supported seamless navigation in the data annotation process. Moreover, partnering with Kili enabled the company to extend its labeling capabilities with professionals trained in the project topic. Using reviewing features on Kili also made collaboration between the internal data scientists and these external professionals fluid and easy to manage. On top of that, Kili stood out among other tools as it was the most convenient tool to iterate with humans in the loop process for the correction and reclassifications.

It was very easy to iterate the labeling process to correct the work of our AI when it was wrong. We were able to import predictions by the AI very quickly to the labeling interface, and the features and the shortcuts within Kili allowed us to modify them easily.

– Vincent, Artificial Intelligence Director.

Transforming the insurance claim filing procedure with artificial intelligence has brought the company visible benefits in terms of productivity and efficiency. Leveraging NLP, the company has simplified the claim form processing to become a quick one-step procedure. Speed was one of the highlights of the result of the transformation, as the usual weeks of procedures became only 15 minutes – which was the time taken to fill out the comprehensive insurance claim form. Identity verification and claim reviews were done in real-time by the AI.

We help customers to build AI that matters across use cases

They trust us for their data-centric journey

Get Started