Automating defect detection with AI in manufacturing company

Transforming car damage assessment in auto-insurance claim process

Scaling up banking customer service with chatbot

Increasing productivity of banking customer email processing

Improving productivity and efficiency in insurance claim form processing

Accelerating retinal disease detection at low cost

How HGH brings trust and performance into its AI video monitoring solutions

Enabling innovative accurate diagnostics for early bladder cancer detection

How Jellysmack improved their NLP and Video models’ performance x5

Ready to see how it works? Begin your AI journey at the click of button.

Get started

How Eidos-Montréal uses machine learning to leverage the Voice of the Customer and make better strategic decisions

How Eidos-Montréal uses machine learning to leverage the Voice of the Customer and make better strategic decisions Eidos-Montréal is a Canadian video game developer based in Montréal, Canada. The studio, created in 2007 and counting 400+ employees today, is responsible for developing some of the most popular video games of all time including the Deus Ex series, the recent Marvel’s Guardians of the Galaxy, Shadow of the Tomb Raider, and many more.

The Eidos-Montréal Analytics and Data Science department were created in 2018 with a set of clear and ambitious objectives: to evangelize the value of analytics services and the use of data in making business and design decisions. Today, the team provides a source of invaluable insights into what customers and fans think about the games developed by the studio, helping drive the Eidos-Montréal production support department’s strategy and development for their current and future games.

Bringing machine learning to the heart of the business

When Marie de Léséleuc, the Data Science and Analytics Director at Eidos-Montréal, created the department of the same name, her objective was set: to bring data and analytics into the conversation and help Eidos-Montréal make more informed decisions both in a game design and business level.

To do so, they decided to collect customer and journalistic reviews and articles, with the goal to understand what is being said about their games and competitors and turn this information into actionable insights. It was decided that a Machine Learning system would be the right tool to do it.

The Data Science team at Eidos-Montréal started collecting thousands of reviews and articles from Steam, Metacritic, and many other specialized websites. Then, they defined a protocol to classify these reviews and articles. The first data point is the articles and review’s topic (what specific topic is being commented about a game) and the second data point is the sentiment associated with it. The team broke down topics into 18 categories including aesthetic, AI, animation, audio, combats, puzzles, performance, playtime, etc.

For sentiment analysis, they defined three types: positive, neutral, and negative. Then, after a long and thorough analysis of thousands of reviews and articles, they created a list of keywords around these categories. These keywords are used to classify any sentence from a review and, or article, determine which category the sentence belongs to, and identify the sentiment associated with it.

The team leveraged a pre-trained model that knows about general NLP categorization and sentiment as a base for their Machine Learning project. Of course, they had some work to do to fine-tune the model to fit the gaming industry and the specific terms used by their customers to talk about games. Still, it is interesting to note that they did not have to start from scratch. Indeed, more and more models are readily available online, a trend in the ML world. Once their dataset and model were ready, the next phase of the project consisted in training the model to get accurate and consistent results.

Training models on a large volume of high quality, unstructured data 

To make their Machine Learning model accurate, it needed to be trained on thousands of reviews and articles. Furthermore, text, especially the customer’s voice, is complex data to analyze. The model needs to recognize the subtle distinction between negative words that mean a positive sentiment, identify sarcasm, and many other forms of language that are specific to the gaming industry, countries, and cultures.

At first, the team chose to train their model themselves. They used Doccano, an open-source annotation tool for machine learning practitioners. It enabled them to create the labeled data they needed to start training the model. To do the annotations, they relied on the willingness of everyone at Eidos-Montréal.

Also, relying on internal resources to annotate hundreds, if not, thousands of reviews and articles were just not scalable, and the data science team had to spend a lot of time reviewing the quality of the annotations.

The challenges of relying on open-source tools and in-house labeling:

This is why they started looking for a more professional approach. After looking at various providers, they decided to run a POC with Kili to test the solution on their use case.

“It appeared quickly that Kili would be able to answer all our needs for our use case. On top of this, the availability of the customer support team made Kili’s integration into our ML workflow painless and quick.” -Marie de Léséleuc, Data Science and Analytics Director at Eidos-Montréal 

High-quality training data to provide actionable insights

For Eidos-Montréal, the key to the success of their project relied on the quality of the annotations at scale, so their model could provide accurate insights to the different business departments.

Boosting efficiency and scalability with external workforce and quality control.

To help with this, the data science team decided to use a professional workforce provided by Kili as part of Kili’s service agreement and partnerships.

“The labelers that Kili provided to support our project are very good. They are familiar with the jargon of the gaming industry, and they understand very well what we expect of them. Also, Kili’s support team made sure to train them on the platform. This accelerated considerably our progress and improved the quality of the annotations.” - Marie de Léséleuc, Data Science and Analytics Director at Eidos-Montréal 

The other key factors for the team were the tools dedicated to quality control that Kili offered on its platform. As Nika explains it, one of the most important features for them was the consensus feature. By leveraging consensus, they make sure to get consistent labels across all their training datasets. It also helps the labelers to improve their skills much faster.

“The consensus feature on Kili’s platform is exactly what we needed. It made a tremendous amount of difference to control the quality of the labelers’ work and get to a level of quality and consistency at a scale that we could not achieve before.” - Nika Abedi, Data Scientist at Eidos-Montréal 

Autonomy and support accelerate project delivery

The data science team at Eidos-Montréal is small. Being autonomous and dedicating their time to value-adding activities is very important to them. By leveraging Kili’s API, they easily add new data to the platform to add them to the labeling queue or create new projects.

The team also widely credits the easy and intuitive Kili interface to create and follow up on projects, without needing any help. The dashboard provides all the information. They need to know; how many annotations were completed, what’s left in the queue, and what is ready for review.

“Kili’s documentation is also very helpful. On various occasions, it helped me quickly solve issues I encountered without having to contact the support team.” - Nika Abedi, Data Scientist at Eidos-Montréal 

“Our understanding of how to ship highly efficient ML models is maturing and Kili plays a great role in this process. With Kili, we are in control of our datasets and the quality of the training we give to our models. The platform also enables us to quickly and easily iterate and ship new projects.” - Marie de Léséleuc, Data Science and Analytics Director at Eidos-Montréal 

Empowering business and devs decision-makers with reliable data

The outcome of this investment is a source of invaluable insights coming straight from their customers, that empowers the teams across the business and development departments to make strategic and operational decisions. The data-science team built dashboards that filter the data and provide analysis of what customers think regarding many aspects of their strategy and products. Recent insights gathered by the tool were compared with social media research collected by the performance marketing team at Eidos-Montréal, which showed a consensus on the findings - enabling the teams to confidently present the intelligence to leadership.

“This initiative is a real success for us. We deliver valuable insights about market trends to our strategy teams. We help marketers better understand what our customers and the market like or dislike about our products and the competition. We support our developers by providing them with direct feedback from our customer base about very specific features in our games.” - Marie de Léséleuc, Data Science and Analytics Director at Eidos Montreal 

With such rapid success, the team is rightfully eager to evolve its capabilities. Ultimately, the team’s roadmap is to keep scaling by always analyzing more reviews and articles, adding more granularity, and opening new sources such as more countries and languages. Then, their goal is to build an automated, continuous improvement floe to train and run their models, all in the cloud for maximum agility, from scraping the data to running the model and providing actionable and valuable insights to the teams.

Gaming - Voice of the Customer

Eidos

They trust us for their data-centric journey

Sed neque, malesuada vitae, nisi. Lorem habitasse dui sit faucibus imperdiet nec. At laoreet id non nunc.

Get Started

Defense - National border surveillance

Defense - Defense communication

Government - Natural disaster management

Retail - In-store cashier-less transactions

Retail - In-store security monitoring

Social media - Brand digital reputation management

Autonomous driving - Road sign and cross mark detection

Agriculture - Crop weed detection

Utilities - Solar network asset maintenance

Government - Public health monitoring

There are currently no events, but check back soon as new events are added all the time.

We help customers to build AI that matters across use cases

They trust us for their data-centric journey

Get Started