We value high-quality data and build the right tools to deliver it

The foundation model era hasn't solved the data problem—it's made it more critical

MNIST's dataset has an error rate of 3.4% and is still cited by more than 38,000 papers. ImageNet, with its crowdsourced labels, has a 6% error rate and underpins the most popular image recognition systems from Google and Facebook.

These errors have real consequences. Models trained on flawed data learn those errors, leading to false predictions in production. The rise of foundation models like GPT-4 and Claude means we're now fine-tuning on domain-specific data and evaluating model performance with business-critical examples. The quality of this data matters more than ever.

Every industry understands AI's transformative potential. But while model architectures have reached commodity status, the differentiator between businesses that succeed with AI and those that don't is data quality: what data trains, fine-tunes, and evaluates your models, how it's created, and how it's governed.

Core Principles

1. Data Quality is critical

Reducing or eliminating labeling errors, getting the right annotations the first time, and focusing on the final input to good machine learning models are now paying huge dividends.

However, data quality can be the most difficult part of developing a reliable model. This is because there is a need for coordination between human intelligence, modeling expertise, project management, and the technology that binds them all together.

This can oftentimes be a painful endeavor. The real differentiator between businesses that are successful at AI and those that aren’t is data quality: what data is used to train & test the algorithm, how is it gathered and labeled, and how is it governed? Our customers’ experience and our experience is that the move to Data-Centric AI (DCAI) is the most important shift businesses need to make today.

‍2. Data Quality is priceless

Human-labeled data is becoming the fuel and compass for AI-based software systems. But the increasing focus on the scale, speed, and cost of building and improving datasets has impacted the data's quality and thus the models' quality.

We have seen reasons for concern first-hand: fairness and bias issues in labeled datasets, quality issues in benchmark datasets, benchmark limitations, reproducibility issues in machine learning research, lack of documentation and data replication, and unrealistic performance measures.

3. Data Quality is complex

While the quality of datasets remains everyone's primary concern, the way it is measured in practice is poorly understood and sometimes just plain wrong.

Data quality is complex—it is not just software bugs or human errors. It is typically the result of how well the annotation is done, how well a dataset and annotation ontology represents the actual task, and if the quality metrics that are available, are suitable for the job.

Data annotation is complex because there are multiple interpretations of the truth, because some gestures are hard, and because collaboration induces complex communication and synchronization.

The development of tools to make repeatable and systematic adjustments to datasets has lagged.

At Kili Technology, we want to reverse this and find new and systematic ways to promote seamless interactions between humans and data.

4. Models have to be developed iteratively

When developing a model, labeling and model testing should work at the same time to remove the unnecessary trial-and-error time spent on improving the model without having to worry or change inconsistent data.

So, if we want to be cost effective, the model development infrastructure must be tightly integrated with a supervision layer so that labeling, model training, and model diagnostics can work in parallel and directly influence the data used for the AI system.

The future of AI is getting the best out of humans and machines by creating a human-in-the-loop machine learning process, thus dramatically accelerating the set up of reliable AI applications.