Securing AI Systems · Topic 6 of 7
Attacks on Models and Training Data
Spot backdoor, trojan, poisoning, inversion, model theft, supply chain, transfer learning, skewing and membership inference attacks and their controls.
- 6 min
- Data poisoning alters training data; model poisoning alters weights or the training process.
- A backdoor responds to a hidden trigger; a trojan hides a malicious payload inside something that looks legitimate.
- Model inversion reconstructs training data; membership inference tests whether a record was in it.
What it is
Some attacks on artificial intelligence (AI) target the model and its data rather than the application around it. They change what the model learns, steal it, or extract what it knows about its training data.
The exam gives you evidence, such as a strange output pattern, a log or a supply chain detail, and asks you to name the attack and pick a compensating control. A compensating control reduces the risk when the root cause cannot be removed at once, for example adding integrity checks while a poisoned data set is rebuilt.
Placing the evidence on this chain is the fastest way to name the attack.
How it works
Attacks that corrupt what the model learns
| Attack | What happens | Typical evidence | Compensating controls |
|---|---|---|---|
| Data poisoning | An attacker inserts or alters training data so the model learns wrong behaviour | Accuracy drops for one class; unexpected records from one contributor; unexplained data changes | Data integrity controls (hashes, signed sources, provenance), restricted write access, data validation |
| Model poisoning | The model's weights or training process are tampered with directly, including poisoned updates in federated learning | Model file hash does not match the registry; one participant's updates differ sharply | Signed model artifacts, access controls on the registry, robust aggregation of updates |
| Backdoor attack | Poisoning plants a hidden trigger, so the model behaves normally except when a specific pattern appears | Correct behaviour in testing, but a particular phrase, sticker or token always produces the same chosen output | Provenance checks, retraining from clean data, testing with unusual inputs |
| Trojan attack | Malicious behaviour or code is hidden inside a model or component that appears legitimate | A downloaded model file runs unexpected code when loaded, or a component behaves maliciously on a trigger | Load only safe formats, scan model files, isolate model loading, approved sources only |
| Introducing biases | Data or feedback is manipulated so the model treats some groups or topics unfairly | Outcomes shift against one group after a data refresh | Bias audits, data balancing, review of new data |
| Model skewing | An attacker skews the distribution of training data, typically by feeding false data into the feedback loop that drives retraining, steering the model toward their goal | Gradual change after a burst of coordinated ratings or reports | Verify feedback before retraining, rate limits on feedback |
The terms backdoor and trojan overlap, and many sources use them as synonyms. A useful way to separate them: a backdoor is the hidden trigger-and-response learned by the model, and a trojan is the disguised malicious payload inside something that looks legitimate. When a question offers both, pick backdoor for a learned trigger and trojan for a disguised malicious file or component.
Attacks that steal or extract
| Attack | What happens | Typical evidence | Compensating controls |
|---|---|---|---|
| Model theft | The model is copied, either by stealing weight files or by sending many queries and training a copy on the answers (model extraction) | Very high query volumes covering the input space; unusual downloads from the registry | Rate limiting, access controls, encryption of stored weights, watermarking, monitoring for extraction patterns |
| Model inversion | Outputs are used to reconstruct sensitive features of the training data | Repeated queries that refine toward one person's attributes | Limit output detail such as raw scores, rate limiting, differential privacy in training |
| Membership inference | An attacker works out whether a specific record was in the training set | Queries with near-identical records comparing confidence levels | Reduce overfitting, return labels rather than exact confidence, differential privacy |
| Output integrity attack | The model's output is altered after it is produced, before it reaches the user or system | Logged model output differs from what the application received | Signed or integrity-checked responses, encryption in transit, secure output pipeline |
Attacks through the supply chain
| Attack | What happens | Compensating controls |
|---|---|---|
| AI supply chain attack | A third-party model, data set, library or plug-in is compromised before you use it | Approved sources, software bill of materials (SBOM) and AI bill of materials listing models and data, signature checks, dependency scanning |
| Transfer learning attack | A malicious pre-trained base model carries hidden behaviour that survives fine-tuning into your model | Vet base model publishers, test the fine-tuned model for backdoors, prefer trusted model hubs with provenance |
Input manipulation
Input manipulation, also called an adversarial example or evasion attack, makes small, deliberate changes to an input so the model misclassifies it. An example is a malware sample padded with harmless-looking features to pass a classifier.
Controls are adversarial training on such examples, input validation, ensembles of different models, and a non-AI detection layer such as signatures.
A worked example
A malware classifier passes all tests, but threat intelligence reports that files containing one unusual string are always rated clean. The model was fine-tuned from a public base model, and part of its training data came from an open submission portal.
The evidence fits a backdoor, introduced either by poisoned submissions or a malicious base model. As compensating controls, the team adds a signature scan for the trigger string, restricts the portal to verified contributors, and retrains from verified data on a vetted base model.
How the exam asks about this
- "A model gives normal results except when inputs contain a specific rare phrase, which always produces the same approval. Which of the following attacks is most likely?" A backdoor attack.
- "Logs show one account sent millions of systematically varied queries to a pricing model's endpoint. Which of the following is the most likely goal?" Model theft through extraction, best countered by rate limiting and monitoring.
- "An attacker can tell whether a patient's record was used to train a diagnostic model. Which of the following attacks is this?" Membership inference.
What to remember for the exam
- Data poisoning alters training data; model poisoning alters weights or the training process.
- A backdoor responds to a hidden trigger; a trojan hides a malicious payload inside something that looks legitimate.
- Model inversion reconstructs training data; membership inference tests whether a record was in it.
- Model skewing distorts training data, usually through the feedback loop; output integrity attacks alter output after the model produces it.
- Supply chain and transfer learning attacks arrive through third-party models, data and libraries.
A backdoored model passes normal testing. Clean test results do not prove a model is safe if its data or base model came from untrusted sources.
Common mistakes
Confusing model inversion with membership inference. Inversion rebuilds what the data looked like; membership inference only answers yes or no for one record.
Assuming fine-tuning removes inherited problems. Backdoors in a base model can survive fine-tuning, which is the whole point of a transfer learning attack.
Choosing a prompt firewall for training-time attacks. Poisoning and backdoors happen before deployment. Data integrity, provenance and access controls address them.
Check yourself
What evidence suggests model extraction?
A very high volume of systematically varied queries from one source, as if mapping the model's whole input space.
How does a transfer learning attack reach a model?
Through a malicious pre-trained base model whose hidden behaviour survives fine-tuning.
Which control helps against both model inversion and membership inference?
Differential privacy during training, along with limiting how much detail outputs reveal.
Topic 6 of 7 in Securing AI Systems
Test yourself on Securing AI Systems
The full CompTIA SecAI+ CY0-001 question bank has 654 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by CompTIA. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.