Challenge
Eidos-Montréal faced a critical challenge in understanding what customers and critics were saying about their games and competitors. As Marie de Léséluc, Data Science and Analytics Director, created the department, her objective was clear: bring data and analytics into strategic conversations to help Eidos-Montréal make more informed decisions at both game design and business levels.
The data landscape was complex. To achieve their vision, the team needed to collect and analyze customer reviews and journalistic articles. The goal was to understand customer sentiment and competitive positioning, then transform this information into actionable insights through a machine learning system.
Training models on high-quality, unstructured data presented significant obstacles. Text data, particularly customer voice, is inherently complex to analyze. The model needed to recognize subtle distinctions—differentiating between negative words used positively, identifying sarcasm, and understanding language forms specific to the gaming industry across different countries and cultures.
The initial approach proved inadequate. The team started with Doccano, an open-source annotation platform, but quickly encountered scalability issues. Relying on internal resources to annotate hundreds or thousands of reviews and articles was simply not scalable. Additionally, the data science team spent considerable time reviewing annotation quality rather than focusing on model development.
"It appeared quickly that Kili would be able to answer all our needs for our use case. On top of this, the availability of the customer support team made Kili's integration into our ML workflow painless and quick."
The success of the entire project hinged on one critical factor: quality annotations at scale. Without accurate, consistent training data, the model couldn't provide reliable insights to business departments.
Solutions
Eidos-Montréal partnered with Kili Technology to build a centralized, scalable platform that could handle their complex natural language processing requirements while ensuring annotation quality and accelerating project delivery.
Centralized Training HubKili provided a single, scalable platform to address all use cases and data types. This eliminated the fragmentation of their previous approach and created a unified environment for managing multiple annotation projects.
Advanced Quality Control Through ConsensusThe consensus feature became essential to the project's success. By having multiple labelers annotate the same assets, the team achieved consistency at scale that was previously unattainable. As Nika Abedi, Data Scientist at Eidos-Montréal, explained, the consensus approach helped maintain quality control and accelerated labeler skill development.
"The consensus feature on Kili's platform is exactly what we needed. It made a tremendous amount of difference to control the quality of the labelers' work and get to a level of quality and consistency at a scale that we could not achieve before."
Professional Labeling WorkforceRather than relying solely on internal volunteers, the data science team leveraged Kili's professional workforce. These labelers were familiar with gaming industry terminology and received direct training on the platform, significantly accelerating progress and improving annotation quality.
"The labelers that Kili provided to support our project are very good. They are familiar with the jargon of the gaming industry, and they understand very well what we expect of them. Also, Kili's support team made sure to train them on the platform. This accelerated considerably our progress and improved the quality of the annotations."
Comprehensive Classification FrameworkThe team developed a sophisticated protocol to classify reviews and articles across two key dimensions:
- Topic categorization: 18 categories including aesthetic, AI, animation, audio, combats, puzzles, performance, and playtime
- Sentiment analysis: Three types (positive, neutral, negative)
After extensive analysis of thousands of reviews and articles, they created a keyword list that enables the model to classify any sentence, determine its category, and identify associated sentiment.
Streamlined Project ManagementKili's intuitive interface and API enabled the team to work autonomously. Using the API, they easily added new data to the platform, managed labeling queues, and created new projects. The dashboard provided complete visibility into project status—tracking completed annotations, queue status, and items ready for review.
Seamless IntegrationThe platform integrated smoothly into their existing machine learning workflow, leveraging pre-trained models for general NLP categorization and sentiment analysis, then fine-tuning for gaming industry-specific terminology and customer language patterns.
Outcome
Eidos-Montréal's AI-powered sentiment analysis system transformed how the company extracts and leverages customer insights, creating measurable value across business and development teams.
Reliable Business IntelligenceThe data science team built dashboards that filter analyzed data and provide clear insights into customer opinions across many aspects of their strategy and products. These insights now directly inform strategic and operational decisions across business and development departments.
Validated Market IntelligenceRecent insights from the tool were compared with social media research conducted by the performance marketing team, showing consensus on key findings. This validation enabled teams to confidently present intelligence to leadership, strengthening data-driven decision-making.
"This initiative is a real success for us. We deliver valuable insights about market trends to our strategy teams. We help marketers better understand what our customers and the market like or dislike about our products and the competition. We support our developers by providing them with direct feedback from our customer base about very specific features in our games."
Enhanced Model MaturityThe team's capability to ship highly efficient ML models continues to advance, with Kili playing a central role in the process. The platform gives them control over dataset quality and model training, enabling rapid iteration and deployment of new projects.
"Our understanding of how to ship highly efficient ML models is maturing and Kili plays a great role in this process. With Kili, we are in control of our datasets and the quality of the training we give to our models. The platform also enables us to quickly and easily iterate and ship new projects."
Scalable Future RoadmapWith their successful foundation, the team is expanding capabilities by analyzing more reviews and articles, adding granularity to their classification system, and incorporating new data sources including additional countries and languages. Their ultimate goal: build an automated, continuous improvement pipeline to train and run models in the cloud for maximum agility—from data scraping to model execution to delivering actionable insights.
Cross-Functional ImpactThe insights now flow to multiple stakeholders: marketers gain deeper understanding of customer preferences and competitive positioning, while developers receive direct, specific customer feedback about game features—creating a direct line from player voice to product development.
.png)
.png)