← 返回列表

DataK9: Auto-categorizing an exabyte of data at field level through AI/ML

单站 Uber Security Engineering 入库 2026-11-18T16:46:38

DataK9: Auto-categorizing an exabyte of data at field level through AI/ML

Sr Staff Engineer, TLM

Introduction

Data categorization–the process of classifying data based on its characteristics and essence–is a foundational pillar of any privacy or security program. The effectiveness of fine-grained data categorization is pivotal in implementing privacy and security controls, such as access policies and encryption, as well as managing the lifecycle of data assets, encompassing retention and deletion. This blog delves into Uber’s approach to achieving data categorization at scale by leveraging various AI/ML techniques.

Why auto-categorization?

Given the monumental engineering cost and the impracticality of manual categorization through decentralized efforts, our experiences have led us to prioritize auto-categorization. In response, we have introduced a novel solution named DataK9, an automatic categorization platform for Uber data. The primary objective is minimizing and eliminating user involvement, thereby addressing the challenges posed by scale, cost, and data owner engagement.

Challenges

Consider a table with a single column comprising purely numeric content. Merely examining the data itself makes it impossible to discern the nature of these numbers, which could represent latitude, longitude, exchange rates, and more.

However, if we are privy to the column name, such as Latitude and Longitude in Figure 1, it significantly aids in categorizing these columns accurately.

Nevertheless, challenges persist, particularly when the column name is generic, as illustrated by the recently added Note column in Figure 2. This could potentially contain information ranging from balance and cost to the distance between two locations.

Moreover, even within the same type of location data, categorization can be nuanced based on precision. For instance, a location with latitude and longitude specified to three or more decimal places may be deemed a precise location. If linked to personal identification, it could warrant classification under the Highly Restricted category.

Let’s delve into another example in Figure 3:

In the absence of explicit context, it becomes challenging to differentiate between personal and business addresses, despite both being addresses.

Given these complexities, there is no universal solution to address all categorization challenges. Therefore, we have opted to leverage a probabilistic-based approach (i.e., AI/ML) to provide assistance in this endeavor.

Strategy

The application of AI/ML techniques forms the bedrock of our auto-categorization initiative. In this section, we present the overarching strategy, delineating the broader flows and dependencies across various systems within the new framework.

Evaluating the efficacy of the new automation process is critical and hinges on a small fraction (\<1%) of labeled (golden) datasets. These datasets are categorized manually, preferably by domain experts or owners. The auto-categorization mechanism then leverages this vetted training data to tag the remaining dataset (>99%). In essence, the solution adopts a hybrid approach, combining manual categorization with automation, encapsulated in three interconnected stages illustrated in the diagram Figure 4 below.

Building Baseline

We suggest categorizing \<1% of datasets (nearly 1,000) as critical and highly utilized tables by humans with close supervision from privacy and domain experts. Those datasets are treated as the “golden” datasets, with high categorization accuracy, and are used to measure automation accuracy. Although the number of “golden” datasets is relatively small, there are some significant benefits:

The selection of datasets for labeling should ideally be done randomly to avoid, as much as possible, statistical biases inserted into the labeled datasets. Such statistical biases reduce the reliability of offline model performance testing, and may cause systemic classification errors in production.

Training

For the rest of the 99% of datasets (>400K), we will primarily utilize a new automation system called DataK9 (detailed in the later sub-section). At first, DataK9 is trained against the manually categorized (golden) datasets until the accuracy reaches a satisfactory level without over-tuning. In particular, there will be two sub-stages:

Serving in Production

After Datak9 demonstrates its satisfactory metrics (i.e., accuracy >90%, F2-score >85%) for the training datasets, it will be ready for production tagging of the rest datasets. However, we plan to take some defensive processes at the early stage of mass categorization. It includes automatic ticket creation to review (preferable for data owners or trained privacy experts) after a dataset is (tentatively) auto-categorized. When the owner stamps it, the categorization will be deemed as final. In addition, during this transition phase, we will closely monitor the progress, most notably the miss-categorization rate. If the feedback from the reviewers is acceptable, we will go for mass deployment of hundreds of thousands of datasets.

Architecture

This section describes the architecture of the AI-based core platform and how DataK9 adopts ML/AI technologies in different parts and stages of the framework as illustrated in Figure 5. We focus on the primary use case of tagging a dataset in this section. It starts with identifying and gathering the essential features of a dataset, followed by the ML/AI technique to determine a column’s weight/score for a specific tag. Then, we discuss how to combine multiple signals for a particular field into the final decision. At last, we detail the active learning process to tune the rules and ML model based on the mistakes made in the original tagging.

Key Features

DataK9 consumes the following information of a dataset that we could use as a potential feature in our various ML/AI classification techniques:

Matching Strategy

The essence of the proposed solution is to crawl data (table or directory, or file) eventually from all data stores and scan each data element periodically to get signals for different tags. We discuss some fundamental matching techniques in this section.

Rule-Based AI

As we crawl a new dataset, we apply the two AI approaches against the sample data and then identify each field/column category. We explain the rule-based in this section, followed by the learning-based in the next section.

We will start with a set of hand-crafted rules and build an associated rule language and engine. It is challenging to find a domain expert with the proper knowledge and experiences, such as a deep understanding of data contents, categorization internals, and engineering chops. Essentially designing a rule considers two critical dimensions: 40+ different tags and fundamental features. We support the following building blocks to express a rule:

Rule Language

Defining a rule is the first step for rule-based AI, and supporting a standard rule definition language or template is imperative. This section describes the essential building blocks to define a rule for a specific tag.

  1. There are three broader sub-rules in a rule depending on the type of data it checks: column value match (columnValueMatch), metadata match (columnNameMatch), and contextual match (contextMatch).
  2. Each sub-rule will have a matching “score” specified by the rule admin based on her domain expertise.
  3. columnValueMatch rule mainly checks the values of each data element in a dataset. These operations are costly and involved because the number of data elements and the number of checks per value is enormous. It entails the following filter and checks:
  4. The bloom filter section contains the configurations required to find the pre-calculated bloom filter to check the membership and the associate score. We can store the bloom filter in a file or table.
  5. Dictionary match looks for the exact match from a predefined list of most commonly used values. This dictionary can be an HDFS file or Hive table.
  6. lengthRange defines the length of the content. This filter can easily exclude a lot of values from expensive matches early. For example, the length of a latitude value could be between 3 to 10. K9 should not consider any content outside this length range for location data.
  7. valueRange specifies the range of valid values of a tag. For example, a person’s age could be between 0-125. If a column contains data beyond this range, K9 will skip this column for any “age” type check. Like lengthRange, valueRange also short-circuits a lot of complex matches.
  8. Pattern match specifies the regular expression to match with the typical pattern of a value. For example, emails always follow a pattern that we can use to check if a column value could be a potential email.
  9. columnNameMatch rule expresses the matching of column name pattern and column type. It could also mention the excluded column names and types that help to avoid false positives.
  10. ContextMatch rule looks for specific contexts that are beyond a particular column value. Currently, DataK9 supports two different contexts with respective matching scores:
  11. resourceContext specifies if there is any table or database name pattern that we want to get a signal. For example, if a database name contains “finance,” there is a high possibility of having transaction data in that dataset.
  12. categoryContext outlines what other associated categories could give additional signals. For example, a level 1 location data tag must have an identifiable personal category in the same dataset.
  13. DataK9 manages the rule’s definitions in one or more YAML files. Alternatively, DataK9 will support rules stored in a database table. The table would be better for dynamically updating the rules or the associated scores.

We present a set of sample rules for “location latitude” to identify a tag in YAML format in Figure 6, which will help understand the concept described in the earlier section. The inline comments in green color entail what each key-value pair means.

Learning-Based AI

Classification is very well studied in the AI/ML world. Our categorization problem fits well for supervised learning for multi-label classification. Using the standard classification algorithm, we intend to use both metadata and the cell content to predict the actual tag. The key idea is shown in Figure 7 and described below:

Aggregating Signals

As described above, DataK9 produces multiple signals with associated score values for each column. A column could have multiple potential tags along with scores estimated by the following methods:

After collecting those signals, DataK9 combines all scores per tag, per column using a weighted aggregation method. The weights of each match are determined empirically and parameterized as part of the rule definition. The combined score determines the final tag after comparing it with pre-specified threshold values. Later versions of the product may use ML-based ensembling methods.

Learning: Feedback Loop

DataK9 heavily relies on ML/AI techniques that could make potential mistakes in label prediction. We intend to auto-learn to avoid similar mistakes by adopting a feedback loop. This section briefly describes how we fit this approach into our framework.

The first part of the challenge is identifying the mistakes. After the auto-tagging, the owner or privacy expert can modify the category using the provided UI. We will keep track of the audit trail of all these modifications in a central location. The next step is to find the typical mistake pattern that could auto-adjust the AI/ML platform. In particular, we strive to adapt our rule database, where we employ many parameters specified empirically. We plan to modify those parameters based on the patterns of mistakes. In addition, we want to alter the different model training parameters based on learning from errors. While our eventual goal is fully automatic, we will start with a human-in-the-loop to adjust the rules. We will eventually move towards an automated feedback loop based on our real-world understanding.

Measuring the Accuracy

In the realm of auto-categorization, accuracy reigns supreme, and thus, we’ve made it our priority to meticulously measure and disclose various levels of metrics tailored to our audience’s needs. At the pinnacle of our reporting hierarchy, we shall unveil the following key metrics:

Standard Metrics for Classification: The quality of our automation frameworks is critical to evaluating and tracking the progress. However, it is impossible to measure the quality of auto-categorization without a proper baseline. Hence, we also proposed that manual categorization of \<1% of datasets covering all data tags by owners/experts would provide a realistic baseline. In the ML/AI world, classification is well-studied literature, and the experts defined a set of metrics to gauge the quality of a classification method. We show them in Figure 8 with a confusion matrix and describe them below:

  1. Accuracy: Classification accuracy is the total number of correct predictions divided by the number of predictions made on all datasets. In other words, this represents what percent of columns are appropriately tagged by machines for the labeled datasets. For instance, if we have 100 columns and K9 categories 90 of them correctly, the accuracy would be 90%. We cannot solely rely on accuracy, as columns with PII are a small percentage of all columns. A classifier that fails to classify anything would have an unusually high accuracy as the true negatives would outweigh the false negatives.
  2. Precision: Precision is how “correct” our positive predictions are. The fewer false positives, the better our precision. However, it does not tell the full story. For example, if we make a single prediction with a 100% success rate, but fail to categorize the other 9 sensitive columns, we have 100% precision, but low accuracy and low recall.
  3. Recall (sensitivity): Recall is how likely we are to identify sensitive information. The fewer false negatives, the better our recall. However, it does not tell the full story. For example, if we predict all columns are sensitive information, we will get 100% recall, but low accuracy and low precision.
  4. F2 Score: The F2 score is a way to skew towards optimizing recall over precision in a single measurable metric.

Precision vs. Recall

The impact of precision and recall would affect two different audiences. For example, low-precision metrics would be concerning for engineers and data scientists. Higher false positives would unnecessarily restrict access to non-sensitive data or delete them prematurely. If there are a lot of false negatives, the system might not be able to restrict sensitive data and enforce appropriate retention that would potentially violate the compliance contracts.

In addition to the above metrics, we strive to measure some additional metrics to track the engineering progress. These metrics measure the following perspectives:

  1. Automation quality: We want to measure how many auto-tags are overturned by humans such as data owners/admins.
  2. Scale: As we have to tag hundreds of thousands of datasets, we need to measure how many datasets we can onboard per day.
  3. Re-classification: When to re-classify any dataset based on schema changes or when new tags are introduced/updated.
  4. Efficiency: This initiative is based on data crawling, which is computationally expensive. We will track how much Uber pays to automate each data element (i.e., columns or tables).
  5. Operation Excellence: Once the development process and initial onboarding are over, we will track how much operations overhead is needed, such as support, bug fixing, and on-call.
  6. Storage System Coverage: As mentioned above, we have different storage technologies and backbones; tracking how many storage instances are onboarded to the categorization effort will be essential.

Production Experiences

Centralized Data Collection System

Uber employs a diverse range of storage systems across various infrastructures, each with its own unique characteristics. Some systems adhere to specific schemas, while others do not. Even among schema-based systems, there can be substantial variations in schema structures.

To streamline the categorization of data stored in these diverse systems, we have implemented a robust data collection system. This system samples data from different storage systems and consolidates them into a centralized data lake. Under this unified approach, the sampled data undergoes processing through a standardized workflow.

Key Benefits:

This strategy not only addresses the challenges posed by the varied nature of storage systems, but also enhances efficiency in data processing and management.

Advancements in Accuracy

Since its initial production launch, DataK9 has undergone continuous enhancements in accuracy over the past few years. We employ two key metrics to showcase the overall accuracy of DataK9 as illustrated in Figure 9:

These metrics serve as robust indicators, illustrating the continual commitment to improving accuracy and checking DataK9’s efficacy in meeting both internal privacy standards and the expectations of data owners.

Success Metrics & Funnel Analysis

In our pursuit of success, we have implemented a comprehensive set of metrics to gauge and optimize our automation process. A detailed funnel (see below) has been meticulously designed to facilitate the investigation and identification of gaps at each step. This invaluable tool provides a systematic approach to trace and monitor the overall categorization status, empowering us to make informed decisions and improvements.

Key Benefits:

This meticulous approach to success metrics and funnel analysis reinforces our commitment to continuous improvement and enables us to proactively address challenges within our automation process.

Conclusion

The DataK9 project at Uber represents a pioneering effort to address the challenges of categorizing data at scale and at the field level through the implementation of AI and ML technologies. Recognizing the fundamental role of data categorization for privacy and security initiatives, Uber has undertaken this initiative to automate and streamline the process.

Cover image attribution: “Categorization system 2” by MCruz (WMF) is licensed under CC BY-SA 4.0.

Apache®, Apache Hive, Hive, Apache Spark and Spark are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.

Lei Sun

Sr Staff Engineer, TLM

Lei Sun is a Tech Lead Manager on Uber's Technical Privacy team. His team builds scalable, reliable, efficient infrastructure to protect customer privacy and empower Uber engineers to seamlessly integrate security, compliance, and privacy into their product development lifecycle.

Mohammad Islam

Mohammad Islam is a Distinguished Engineer at Uber. He currently works within the Engineering Security organization to enhance the company's security, privacy, and compliance measures. Before his current role, he co-founded Uber’s big data platform. Mohammad is the author of an O'Reilly book on Apache Oozie and serves as a Project Management Committee (PMC) member for Apache Oozie and Tez.

原文链接:https://www.uber.com/au/en/blog/auto-categorizing-data-through-ai-ml/