Data Management Specific to AI

What it is

A model learns its behaviour from data. If the data is wrong, unrepresentative, biased or used without permission, the model will be too, however good the algorithm. Data management specific to AI covers how training, validation, test and retrieval data is sourced, prepared, labelled, versioned and monitored.

The auditor evaluates data input requirements: is the data appropriate for the purpose, free from unacceptable bias, permitted under privacy law and of documented quality? The outline names these three concerns, appropriateness, bias and privacy, directly.

How it works

Data input requirements

Requirements should be documented before data is collected. A common finding is a model built on whatever data was available, with requirements written later to fit.

Sources of bias in data

Removing a protected attribute from the data does not remove bias if proxies remain. The auditor expects bias testing on outcomes, not just a check that sensitive columns were dropped.

Lineage, versioning and datasheets

Data lineage traces each training dataset back to its sources and transformations. Versioning records exactly which dataset version trained each model version. Without both, the organization cannot reproduce a model, explain a result or remove problem data.

A datasheet (dataset documentation) records purpose, source, collection method, known limitations and approved uses. It is strong evidence that data was understood before use.

Labelling quality

Supervised models depend on labels. The auditor checks labelling instructions, labeller training, quality checks such as a second reviewer on a sample, measures of agreement between labellers, and controls over outsourced labelling vendors.

Data splits and leakage

Data is normally split into training, validation and test sets. The test set must stay separate until final evaluation. Data leakage occurs when information from the test set, or from the future, slips into training, so test results look better than real performance. The auditor checks how splits were made and that the test set was locked.

Retrieval data for generative AI

In retrieval augmented generation, the documents the model searches are a live data input. They need the same governance: approved sources, access controls that respect user permissions, freshness checks and a process to remove outdated or wrong content.

Data drift

After deployment, input data changes. Data drift is a change in the distribution of inputs; concept drift is a change in the relationship between inputs and the correct answer. Both degrade performance. Data management must include monitoring that compares production inputs with the training baseline.

A worked example

A hospital trains a model to flag patients at risk of readmission. The auditor obtains the datasheet and finds the training data came from two urban hospitals, while the model is deployed across rural clinics as well. No representativeness assessment exists, and performance has not been measured by site.

The auditor reports that data appropriateness was not evaluated for the deployed population. The recommendation is to measure performance for rural sites and retrain or restrict use if thresholds are not met.

How the exam asks about this

• "What should the auditor review FIRST to assess the risk of bias in a model?" The training data's sources and representativeness of the population the model serves.
• "Which is the BEST evidence that a model can be reproduced?" Version records linking the model version to the exact dataset, code and parameters used.
• "Test results are much better than production results. What is the MOST likely cause?" Data leakage between training and test data, or drift in production data.