AI Risk Management · Topic 2 of 4
The AI Threat Landscape
AI threats at concept level: poisoning, evasion, extraction, inference, prompt injection, the OWASP Top 10 for LLM Applications and the MITRE ATLAS matrix.
- 5 min
- Poisoning and backdoors attack training data and supply chain; evasion and prompt injection attack inference.
- Model inversion and membership inference are privacy attacks on training data.
- Indirect prompt injection arrives through content the AI reads.
What it is
The AI threat landscape is the set of actors, techniques and weaknesses that can harm AI systems or use AI to harm the organization. It has two sides: attacks on AI, which target models, data and AI applications, and attacks with AI, where adversaries use AI to scale phishing, fraud and deepfakes.
The security manager does not need to perform these attacks. The manager needs to recognize them, know where in the AI life cycle each one strikes, and translate threat intelligence into risk for this organization.
How it works
Attacks on AI by life cycle stage
| Attack | Stage | What it does | Main concern |
|---|---|---|---|
| Data poisoning | Training, fine-tuning, retrieval data | Corrupts data so the model learns wrong or hidden behaviour | Integrity |
| Backdoor | Training or supply chain | Hides a trigger that makes the model misbehave on specific inputs | Integrity |
| Evasion (adversarial examples) | Inference | Inputs crafted so the model misclassifies them, for example to slip past a fraud or malware detector | Integrity |
| Model extraction or theft | Inference or storage | Copies the model's behaviour through many queries, or steals the weights | Confidentiality, IP |
| Model inversion and membership inference | Inference | Infers training data, or whether a record was in the training set | Privacy |
| Prompt injection | Inference | Instructions in input override the system's intended behaviour | Integrity, confidentiality |
| Jailbreaking | Inference | Inputs that get the model to ignore its safety rules | Safety, reputation |
| Resource abuse | Inference | Floods of costly requests that exhaust capacity or budget | Availability, cost |
| Supply chain compromise | Acquisition | Malicious or vulnerable models, datasets or libraries | All |
Direct prompt injection comes from the user. Indirect prompt injection is hidden in content the AI reads, such as a web page, email or document in a retrieval source. Indirect injection is especially dangerous for agents, because the attacker never needs to talk to the system.
OWASP Top 10 for LLM Applications (2025)
OWASP publishes the most widely used list of risks for applications built on large language models. The 2025 edition names:
| Risk | In short |
|---|---|
| Prompt Injection | Inputs alter the model's behaviour against the designer's intent |
| Sensitive Information Disclosure | The model or application reveals confidential or personal data |
| Supply Chain | Compromised models, datasets, libraries or platforms |
| Data and Model Poisoning | Manipulated training, fine-tuning or embedding data |
| Improper Output Handling | Model output passed to other systems without validation |
| Excessive Agency | The AI has more permissions, functions or autonomy than it needs |
| System Prompt Leakage | Hidden instructions, sometimes with secrets, are revealed |
| Vector and Embedding Weaknesses | Flaws in retrieval stores, such as missing access control or poisoned entries |
| Misinformation | False or misleading output that people rely on |
| Unbounded Consumption | Uncontrolled use that causes denial of service or runaway cost |
MITRE ATLAS
MITRE ATLAS is a knowledge base of adversary tactics and techniques against AI-enabled systems, modelled on MITRE ATT&CK. It adds AI-specific tactics such as AI Model Access, where an adversary gains some form of access to a model, and AI Attack Staging, where the adversary prepares an attack, for example by building a proxy model. It also includes real-world case studies and mitigations.
Security teams use ATLAS for threat modelling, mapping detections and planning red team scope. NIST also publishes a taxonomy of adversarial machine learning attacks and mitigations (NIST AI 100-2), useful as a common vocabulary.
Attacks that use AI
| Use by attackers | Effect on the organization |
|---|---|
| Deepfake audio and video | Impersonation of executives to authorize payments or reset access |
| AI-written phishing | Fluent, personalized messages at scale |
| Faster reconnaissance and exploit development | Shorter time between disclosure and attack |
| Synthetic identities | Fraud against onboarding and verification |
Controls here are mostly familiar ones strengthened: out-of-band verification for payment requests, phishing-resistant authentication and updated awareness training.
Internal threats and failures
Not every AI threat is malicious. Employees leak data into public tools, developers connect agents with excessive permissions, and models drift into error. The threat landscape includes these unintentional sources, which are often more frequent than attacks.
A worked example
Threat intelligence reports attackers hiding instructions in documents so that AI assistants that read them email data to outside addresses. The organization runs an email assistant that reads incoming mail and can send messages.
The security manager maps the threat to that asset: indirect prompt injection, combined with excessive agency. Risk is reassessed and treatment is agreed: the assistant may draft but not send external email without user approval, and outbound content is checked for sensitive data.
How the exam asks about this
- "An AI assistant follows instructions hidden in a web page it summarizes. What type of attack is this?" Indirect prompt injection.
- "Which resource BEST supports threat modelling of attacks against AI systems?" MITRE ATLAS.
- "An attacker sends many queries to replicate a proprietary model. What is the MAIN risk?" Model extraction, a loss of intellectual property.
What to remember for the exam
- Poisoning and backdoors attack training data and supply chain; evasion and prompt injection attack inference.
- Model inversion and membership inference are privacy attacks on training data.
- Indirect prompt injection arrives through content the AI reads.
- OWASP Top 10 for LLM Applications 2025 includes Excessive Agency, System Prompt Leakage and Unbounded Consumption.
- MITRE ATLAS extends ATT&CK with AI tactics such as AI Model Access and AI Attack Staging.
Threat intelligence only matters when it is mapped to your own AI assets. A new technique against agents is low priority if you run no agents, and urgent if you do.
Common mistakes
Confusing poisoning with evasion. Poisoning corrupts what the model learns; evasion fools a trained model at inference time.
Thinking prompt injection only comes from users. Indirect injection through documents, emails and web content is often the larger risk.
Ignoring attacks that use AI. Deepfake-enabled fraud targets business processes, not models, and needs process controls.
Check yourself
What is the difference between direct and indirect prompt injection?
Direct injection comes from the user's own input; indirect injection is hidden in content the AI processes, such as documents, emails or web pages.
What does a membership inference attack reveal?
Whether a particular record was part of a model's training data, which is a privacy risk.
What is MITRE ATLAS?
A knowledge base of adversary tactics, techniques and case studies against AI systems, modelled on MITRE ATT&CK.
Topic 2 of 4 in AI Risk Management
Test yourself on AI Risk Management
The full ISACA AAISM question bank has 747 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by ISACA. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.