Banking - Increasing productivity of customer email processing
Impact
Challenges
Creation of a classification scheme of reference
Quality control over consistency in labeling data to ensure accuracy of the model
Need human in the loop to enrich data and track quality
Solutions
Efficient collaboration workflows between data scientists and annotators during the labelling process, making learning and iteration quicker
Advanced quality review features including consensus and honeypot to control of the labeling quality
Context
For years, a major French banking institution was having a bottleneck in inefficiency when it comes to handling emails processing for customer service. A fully manual procedure with human personnel was very time-consuming and costly, and the productivity level per employee was expected to increase.
When the bank decided to proceed with email processing powered by artificial intelligence (AI), it partnered with Kili Technology to train the automation models with high-quality datasets.
How it all started
Realizing the importance of artificial intelligence automation
As one of the leading European banking institutions since the 80s, the bank serves millions of customers worldwide. Every day, the bank receives thousands of emails from customers, be it in form of customer complaints, questions or inquiries over banking products, registrations, document attachments follow up, among others. Regarding this, the bank was facing one problem that led to a significant bottleneck in inefficiency: the time and effort taken to process those emails manually.
Extracting information, verifying the quality, classifying and analyzing thousands of documents, from bank statements, forms, applications, and more costs the banks not only in time but also money. The tasks are time consuming and heavily repetitive, causing the need for more employees to reduce backlogs from occurring
It takes hours for my staff to read the emails and check the attachments, only to forward it to another staff as the email he opened is not relevant to him and so he couldn’t respond directly.” Explained Guillaume, Director of Technology and Innovation at the bank.
Challenge
The challenges of building the AI automation model
Building an artificial intelligence model for email processing is not without its challenges. First of all, it is challenging to create a classification scheme that is simple enough to be understood by the AI model. Second, there should be a strategy that allows the bank to locate risk on mistakes in classification.
When two or more staff read the same email, they might have different agreements on who they should route the email to. This represents the risk of a mistake that the AI model can make. Hence we need to foresee this to make the model accurate and efficient.
Moreover, another challenge is to facilitate the AI automation model to have room for data correction and reclassification of data when there is a mistake in identifying information or attached documents within the email. Human feedback can enrich the model and make it more robust. To address these challenges, the bank then needed to make sure that the AI model is trained rigorously with datasets that are accurately labeled to ensure the high-quality output of performance.
Issues with raw data from banking documentation
Complex layouts
Nested tables
Handwriting
Multiple docs in a single PDF
Symbols
Images, logos, stamps
Noisy images
Contextual relationships (footnotes)
Solution
A versatile training data platform to implement Intelligent Document Processing (IDP)
Kicking off with the project, the bank formed partnerships with companies. Aside from choosing AI consultancy and implementation service, one key strategic partner is a data annotation tool and service. As the bank is already aware, the data annotation process is vital to produce high-quality datasets to train the AI model to be able to avoid mistakes during work. Based on the challenges mapped, the bank was looking for a data annotation company that could provide a simple tool with rigorous quality management and flexibility in collaboration.
We assessed several providers available in the market and tested each of them. Kili Technology stood out as the best solution that could best facilitate our needs
Choosing the most suitable data annotation solution mainly revolved around ROI. The bank needed to assess the cost-benefit analysis very carefully to make sure the return is favorable for them when they invest. In addition, the bank also paid attention to data privacy and the security of sensitive documents.
Working with Kili, the bank found it useful to leverage advanced features such as consensus and honeypot to control the data quality, aside from its advanced technical labeling features. On top of that, the Kili platform also simplifies the collaboration between data scientists and annotators during the labeling process, making learning and iteration quicker. Data security is highly protected and guaranteed.
Efficient collaboration workflows between data scientists and annotators during the labelling process, making learning and iteration quicker
Advanced quality review features including consensus and honeypot to control of the labeling quality
Impact
“It’s the impact that matters”
After the project is implemented, currently 92% of the customer emails are processed with automation. The bank’s customer service employees feel a positive impact on their productivity, as more time can be allocated to more urgent tasks than reading emails and opening attachments.
Solving customer inquiries through email previously took 20 minutes, with automated email classification it takes only 7 minutes – including providing answers and solving the issue for the customer. It saved approximately 14,560 hours of employee work per year in branches in France only. This equates to approximately 10 FTEs. Moreover, human in the loop during the model training increases data accuracy from 80% to 85%, which allows for a significant productivity improvement.
It’s exciting to see the impact. It’s the impact that matters. We don’t only scale up our processes to be extremely efficient and reduce our costs, but we also increase the satisfaction of both customers and employees towards the bank.
Lesson Learned
Labeled data plays a vital role in determining the accuracy of an artificial intelligence model, hence investing time and effort in data labeling is of the utmost importance to ensure the success of the project
AI model accuracy saves cost and time. It saves an equivalent of 10 FTEs hours of work. Human in-the-loop quality management helps accuracy to improve from 80% to 85%, which boosts productivity significantly.
It is important to select a suitable data annotation partner for your AI project. Factors such as robustness, meticulous quality management, and simplicity of collaboration are key.
Related Resources
AI For Compliance: What, Why and How?
Kili’s guide to the new AI paradigm
Opinion Classification with Kili & HuggingFac...
Get Started
Get started! Build better data now.