AI Operations · Topic 6 of 8
Threats and Vulnerabilities Specific to AI
AI threats for auditors at concept level: poisoning, evasion, extraction, inference, prompt injection, OWASP Top 10 for LLM Applications and MITRE ATLAS.
- 5 min
- Poisoning and backdoors strike training and supply chain; evasion and prompt injection strike use.
- Inversion and membership inference are privacy threats.
- Indirect prompt injection arrives through content the AI reads.
What it is
AI systems face the usual IT threats plus new ones that target data, models and model behaviour. Some come from attackers, others from weaknesses in design, such as giving an AI agent more permissions than it needs. AI is also used by attackers against the organization, for example in deepfake fraud.
The auditor needs to recognise these threats at concept level, know where in the lifecycle each one strikes, and judge whether the organization's threat assessment covers them. This page covers the threats; the next page covers how the auditor evaluates the programs that manage them.
How it works
Attacks on AI by lifecycle stage
| Threat | Stage | In short | Main impact |
|---|---|---|---|
| Data poisoning | Training, fine-tuning or retrieval data | Corrupted data teaches wrong or hidden behaviour | Integrity |
| Backdoor | Training or supply chain | A hidden trigger causes misbehaviour on certain inputs | Integrity |
| Evasion (adversarial examples) | Use | Crafted inputs make a trained model misclassify | Integrity |
| Model extraction | Use | Repeated queries copy a model's behaviour | Intellectual property |
| Model inversion and membership inference | Use | Infer training data or whether a record was used | Privacy |
| Prompt injection | Use | Inputs override the system's intended instructions | Integrity, confidentiality |
| Jailbreaking | Use | Inputs get the model to ignore safety rules | Safety, reputation |
| Resource abuse | Use | Costly request floods exhaust capacity or budget | Availability, cost |
| Supply chain compromise | Acquisition | Malicious or vulnerable models, datasets or libraries | All |
Direct prompt injection comes from the user's own input. Indirect prompt injection is hidden in content the AI reads, such as a document, email or web page. Indirect injection is especially serious for agents that can take actions.
OWASP Top 10 for LLM Applications 2025
The OWASP list is a widely used reference for risks in applications built on large language models.
| Risk | In short |
|---|---|
| Prompt Injection | Inputs alter model behaviour against the designer's intent |
| Sensitive Information Disclosure | Confidential or personal data revealed in output |
| Supply Chain | Compromised models, data, libraries or platforms |
| Data and Model Poisoning | Manipulated training, fine-tuning or embedding data |
| Improper Output Handling | Output passed to other systems without validation |
| Excessive Agency | More permissions, functions or autonomy than needed |
| System Prompt Leakage | Hidden instructions or embedded secrets revealed |
| Vector and Embedding Weaknesses | Flaws in retrieval stores, such as missing access control |
| Misinformation | False or misleading output that people rely on |
| Unbounded Consumption | Uncontrolled use causing outage or runaway cost |
MITRE ATLAS
MITRE ATLAS is a knowledge base of adversary tactics, techniques and real case studies against AI-enabled systems, modelled on MITRE ATT&CK. It adds AI-specific tactics such as AI Model Access, where an adversary gains some form of access to a model, and AI Attack Staging, where the adversary prepares an attack, for example by building a proxy model. Organizations use it for threat modelling and to scope red team exercises.
Vulnerabilities that are design weaknesses
| Weakness | Example |
|---|---|
| Excessive permissions | An agent with write access to all records when it only needs to read |
| No output validation | Generated text passed straight into a database query or web page |
| Weak retrieval access control | An assistant shows a user documents they could not open directly |
| Secrets in prompts | Credentials placed in system instructions |
| No rate limits | Unlimited queries enable extraction and cost abuse |
Threats that use AI
Attackers use AI for convincing phishing, deepfake voice and video to impersonate executives, synthetic identities and faster reconnaissance. Controls are mostly process controls: out-of-band verification for payment and access requests, phishing-resistant authentication and updated awareness.
Unintentional threats
Not every AI threat is malicious. Staff leak data into public tools, developers over-permission agents, and models drift into error. These are often more frequent than attacks and belong in the same assessment.
A worked example
An auditor reviews a customer support assistant that reads customer emails and can issue refunds. The threat model lists only external hacking of the hosting platform. It does not consider instructions hidden in incoming emails, and the assistant can issue refunds of any size without approval.
The auditor reports that the threat assessment omits indirect prompt injection and excessive agency. The recommendation is to update the threat model using a reference such as OWASP or ATLAS and to limit refund authority, with human approval above a set value.
How the exam asks about this
- "An assistant follows instructions hidden in a document it was asked to summarise. What threat does this BEST describe?" Indirect prompt injection.
- "Which resource is MOST useful for threat modelling attacks against AI systems?" MITRE ATLAS.
- "An AI agent can delete records although it only needs to read them. Which OWASP LLM risk applies?" Excessive Agency.
What to remember for the exam
- Poisoning and backdoors strike training and supply chain; evasion and prompt injection strike use.
- Inversion and membership inference are privacy threats.
- Indirect prompt injection arrives through content the AI reads.
- OWASP LLM 2025 includes Excessive Agency, System Prompt Leakage, Vector and Embedding Weaknesses and Unbounded Consumption.
- MITRE ATLAS tactics include AI Model Access and AI Attack Staging.
The auditor evaluates whether threats were identified and treated; the auditor does not demonstrate attacks. Authorised testing is done by red teams under agreed scope.
Common mistakes
Confusing poisoning with evasion. Poisoning corrupts learning; evasion fools a trained model at use time.
Treating prompt injection as a user-only risk. Content the AI reads can carry instructions too.
Ignoring permissions. Many AI incidents come from what the system was allowed to do, not from clever attacks.
Check yourself
What is the difference between data poisoning and evasion?
Poisoning corrupts the data a model learns from; evasion uses crafted inputs to fool a model that is already trained.
What does membership inference reveal?
Whether a particular record was in a model's training data, which can expose personal information.
What is Excessive Agency in the OWASP list?
An AI application having more permissions, functions or autonomy than its task needs, so errors or manipulation cause greater harm.
Topic 6 of 8 in AI Operations
Test yourself on AI Operations
The full ISACA AAIA question bank has 750 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by ISACA. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.