Cloud Data Security · Topic 7 of 7
Protecting AI and ML Data and Models
CCSP protection of AI and ML data: data set and model privacy, inference and extraction risks, poisoning, and validating and verifying data and models.
- 5 min
- Training data is personal or confidential data and keeps all its legal duties.
- Validation is fitness for purpose; verification is authenticity and integrity.
- Poisoning attacks the data; membership inference, inversion and extraction attack privacy through the model.
What it is
AI and ML systems turn data into models. Both are assets: the training data set often holds personal or confidential data, and the model can leak that data or be stolen or tampered with. Most training and hosting happens in the cloud, on shared platforms.
The CCSP asks for two things: keep data sets and models private, and keep them secure through validation and verification so that what you train on and deploy is what you intended.
How it works
Data set privacy
Training data is subject to the same rules as any other data, and to the same lifecycle controls.
- Lawful basis and purpose: using customer data to train a model may be a new purpose that needs a legal basis, notice or consent.
- Minimization: use only the fields needed; remove direct identifiers where possible.
- Privacy enhancing techniques: pseudonymization, aggregation, synthetic data and differential privacy, which adds calibrated noise so individual records cannot be singled out.
- Location: training data copied to another region or to a provider's AI service can be a cross border transfer.
- Retention and deletion: a deletion request may affect training data, and it is hard to remove a person's influence from an already trained model, so minimize before training.
Model privacy
A trained model can reveal what it learned from.
| Threat | What happens |
|---|---|
| Membership inference | An attacker learns whether a specific record was in the training set |
| Model inversion | An attacker reconstructs sensitive features of training data from outputs |
| Training data extraction | A generative model repeats memorized text, such as personal data or secrets |
| Prompt and output leakage | Users' prompts and model outputs, stored by the service, expose confidential data |
Controls include limiting the detail of outputs (for example no raw confidence scores), rate limiting queries, output filtering, differential privacy during training, and contract terms stating whether a provider may use customer prompts to train its models.
Data set security: validation and verification
Validation checks that data is fit for purpose: correct, complete, representative and properly labelled. Verification checks that data is authentic and unchanged: it came from the approved source and has not been tampered with.
| Control | Purpose |
|---|---|
| Provenance records | Document source, owner, licence and collection date of each data set |
| Hashes and signatures | Prove a data set version is the one that was approved |
| Access control and versioning | Restrict who can change training data; keep immutable versions |
| Quality and bias checks | Detect missing values, skew and unrepresentative samples |
| Outlier and poisoning detection | Find injected or mislabelled records before training |
Data poisoning inserts crafted records so the model learns wrong or hidden behaviour, such as a backdoor that triggers on a certain input. Strong verification of sources is the main defence.
Model security: validation and verification
Models need the same discipline as software.
- Validation: test accuracy, robustness and fairness on held back data and adversarial inputs before release.
- Verification: sign model artefacts, record their hashes in a registry, and deploy only verified versions.
- Supply chain: pre-trained models and public data sets from third parties can carry backdoors or malicious code in their file formats; scan and test them before use.
- Access: protect model weights as intellectual property with encryption and least privilege; limit query rates to make model extraction, copying a model through repeated queries, harder.
- Monitoring: watch for drift in input data and output quality, which can signal a change in the environment or an attack.
Shared responsibility for AI services
| Service | Customer is responsible for |
|---|---|
| Training on IaaS | Everything above the infrastructure: data, pipelines, model, endpoints |
| Managed ML platform | Data, configuration, access, model validation |
| AI SaaS or model API | Data sent in prompts, settings for data retention and training use, access control |
A worked example
A bank fine-tunes a language model on customer support transcripts in a managed ML service. Testers find the model sometimes repeats real account numbers from the transcripts.
The data was not minimized before training. The bank removes the model, rebuilds the training set with account numbers and names tokenized, records the new data set hash in its registry, and adds output filtering for number patterns. It also confirms in the provider contract that its training data and prompts are not used to train the provider's own models.
How the exam asks about this
- "An attacker determines whether a person's record was used to train a model. What attack is this?" Membership inference.
- "Which control BEST prevents an unapproved training data set from being used?" Verifying the data set's hash or signature against an approved registry version.
- "What is the MOST effective way to stop a model revealing personal data from its training set?" Minimize or de-identify the data before training.
What to remember for the exam
- Training data is personal or confidential data and keeps all its legal duties.
- Validation is fitness for purpose; verification is authenticity and integrity.
- Poisoning attacks the data; membership inference, inversion and extraction attack privacy through the model.
- Sign and register data sets and models; deploy only verified versions.
- Check what an AI provider does with prompts and outputs, including training use and retention.
Removing personal data from a model after training is hard and sometimes impossible. Privacy controls must be applied before the data reaches the training pipeline.
Common mistakes
Treating models as harmless code. Models can memorize and leak data, and their weights are valuable intellectual property.
Trusting public data sets and pre-trained models. Third party AI artefacts can be poisoned or carry malicious code.
Ignoring prompts as data. What users type into an AI service is stored and processed and may be confidential.
Check yourself
What is the difference between data validation and data verification for AI?
Validation checks the data is correct, complete and representative for its purpose, while verification checks it is authentic and has not been altered.
What does model extraction mean?
Copying a model's behaviour by sending many queries and training a substitute on the answers.
Why might using customer data to train a model need a new legal basis?
Training can be a different purpose from the one the data was collected for, so privacy laws may require notice, consent or another basis.
Topic 7 of 7 in Cloud Data Security
Test yourself on Cloud Data Security
The full ISC2 CCSP question bank has 900 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by ISC2. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.