AI Governance and Risk · Topic 4 of 6
Privacy and Data Governance Programs for AI
Auditing privacy and data governance for AI: data ownership, lawful basis, minimization, privacy impact assessments, data subject rights and AI data risks.
- 5 min
- Data governance and privacy programs must explicitly cover training data, prompts, outputs and models.
- Reusing data to train AI raises purpose limitation questions.
- A privacy impact assessment should precede high-risk processing.
What it is
AI depends on data, so the data governance program and the privacy program are part of AI governance. Data governance decides who owns data, what quality it must meet and how it may be used. The privacy program makes sure personal data is processed lawfully, fairly and transparently.
The auditor evaluates whether both programs have been extended to AI. A program written for databases and reports often says nothing about training data, prompts, model outputs or inferences, and that silence is the gap.
How it works
Data governance applied to AI
| Element | What the auditor expects for AI |
|---|---|
| Data ownership | A named owner approves each dataset's use for training or retrieval |
| Data classification | Training data, prompts, outputs and model files are classified like other data |
| Data catalogue and lineage | Sources of training data are recorded and traceable |
| Quality standards | Defined requirements for completeness, accuracy and representativeness |
| Rights to use | Licences, contracts and consent permit the AI use |
| Retention and disposal | Rules cover training sets, logs and prompts, not just source records |
A model can memorise parts of its training data. That makes the model itself a data asset that may contain personal or confidential information, and it should be classified and protected on that basis.
Privacy principles under pressure from AI
| Principle | How AI challenges it |
|---|---|
| Purpose limitation | Data collected for one purpose is reused to train a model for another |
| Data minimization | Teams want as much data as possible for better models |
| Transparency | People may not know AI is used or how it affects them |
| Accuracy | Models can generate false statements about real people |
| Storage limitation | Data stays embedded in models after source records are deleted |
| Individual rights | Access, correction and deletion are hard once data is in a model |
Privacy law such as the EU GDPR also restricts decisions based solely on automated processing that have legal or similarly significant effects on people. Where such decisions are made, the auditor expects safeguards such as human review, a chance for the person to express their view and a way to contest the decision.
Privacy impact assessments
A privacy impact assessment (or data protection impact assessment) is expected when AI processes personal data in a way likely to create high risk, such as profiling or large-scale processing of sensitive data. The auditor checks that it was done before processing started, that the privacy function reviewed it, and that the actions it identified were completed.
Privacy-enhancing techniques
| Technique | What it does |
|---|---|
| Pseudonymization | Replaces identifiers; still personal data if it can be re-linked |
| Anonymization | Removes the ability to identify people; hard to achieve with rich data |
| Synthetic data | Artificial records that mimic real data patterns |
| Differential privacy | Adds controlled noise so individuals cannot be singled out |
| Federated learning | Trains across sites without centralising raw data |
The auditor does not need to verify the mathematics. The auditor checks that the technique was chosen for a documented reason and that its limits, such as re-identification risk, were assessed.
Generative AI and staff use
Staff pasting customer or confidential data into public AI tools is a common privacy failure. The auditor checks the acceptable use policy, technical controls such as approved enterprise tools and data loss prevention, contract terms on whether the provider may use inputs for training, and evidence that staff were trained.
A worked example
A telecom company trains a churn model on call records and customer complaints. The data was collected to provide service and handle complaints. No privacy impact assessment exists, and the data catalogue does not record that the dataset is used for training.
The auditor reports that the use was not assessed against purpose limitation and that data governance did not capture the new use. The recommendation is that the privacy function assess lawful basis and risk, and that the data owner formally approve or refuse the training use.
How the exam asks about this
- "What should the auditor review FIRST before an AI model is trained on customer data?" Whether the use is permitted by the data's lawful basis, purpose and the data owner's approval.
- "Which control BEST reduces the risk of staff exposing personal data in public generative AI tools?" An approved enterprise tool with contractual data protection, backed by policy, data loss prevention and training.
- "A customer requests deletion of their data, which was used to train a model. What is the auditor's MAIN concern?" Whether the organization has a process to address personal data that persists in models and derived datasets.
What to remember for the exam
- Data governance and privacy programs must explicitly cover training data, prompts, outputs and models.
- Reusing data to train AI raises purpose limitation questions.
- A privacy impact assessment should precede high-risk processing.
- Pseudonymized data is still personal data.
- Solely automated decisions with significant effects need safeguards such as human review.
Deleting the source record does not remove what a model has learned. Retention and deletion procedures must address trained models and derived datasets.
Common mistakes
Assuming anonymized means safe. Rich datasets can often be re-identified; check how anonymization was tested.
Treating outputs as harmless. Model outputs can contain or infer personal data and need classification.
Letting the data science team approve its own data use. The data owner and privacy function approve; developers request.
Check yourself
Why is a trained model treated as a data asset?
Because it may have memorised training data, so it can contain personal or confidential information and needs classification and protection.
When should a privacy impact assessment for an AI system be done?
Before processing starts, when the AI use is likely to create high risk to individuals, such as profiling.
What is the difference between pseudonymization and anonymization?
Pseudonymized data can be re-linked to individuals and remains personal data; properly anonymized data can no longer identify anyone.
Topic 4 of 6 in AI Governance and Risk
Test yourself on AI Governance and Risk
The full ISACA AAIA question bank has 750 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by ISACA. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.