Threats, Vulnerabilities, and Attacks · Topic 10 of 10
Threats and Vulnerabilities of AI Usage
Security+ AI risks: prompt injection, jailbreaking, poisoning, evasion, model manipulation, hallucinations, bias, data loss, privacy and code execution.
- 5 min
- Poisoning corrupts training data. Evasion fools a model at use time with crafted input.
- Model manipulation tampers with the model itself.
- Prompt injection overrides instructions, directly or through hidden content. Jailbreaking gets around safety rules.
What it is
Artificial intelligence (AI) systems, especially large language models (LLMs), are now part of everyday business tools. They bring new kinds of weakness, because they learn from data, follow instructions written in plain language and can act on other systems.
Security+ expects you to name each AI threat, say what it is and pick the matching control. This page stays at that level.
Each box in that chain can be attacked. Poisoning targets the data, model manipulation targets the model, prompt injection and jailbreaking target the prompt, and unsafe output handling can lead to code execution.
How it works
Attacks on the data and the model
| Threat | What it is | Main control |
|---|---|---|
| Poisoning | Adding false or malicious data to training or fine tuning data so the model learns the wrong behavior or a hidden trigger | Trusted, verified data sources, data integrity checks, testing before release |
| Model manipulation | Tampering with the model itself, its settings or its weights, or swapping in an untrusted model | Access control on models, integrity checks, use of trusted model sources |
| Evasion | Crafting input that the model misreads at use time, such as a file altered so an AI malware detector calls it safe | Testing with adversarial examples, layered detection that does not depend on one model |
Poisoning happens before or during training, while evasion happens when the model is in use. The data is attacked in one case and the input in the other.
Attacks through the prompt
| Threat | What it is | Main control |
|---|---|---|
| Prompt injection | Input that overrides the system's instructions, either typed directly or hidden in content the model reads, such as a web page, email or document | Treat all model input as untrusted, separate instructions from data, filter input and output, limit what the model can do |
| Jailbreaking | Prompts written to get around the model's safety rules so it produces content it should refuse | Guardrails, output filtering, monitoring and testing of prompts |
Indirect prompt injection is the more dangerous form. The user asks an assistant to summarize a page, and hidden text on that page tells the assistant to send data elsewhere.
Risks in what the model produces
| Threat | What it is | Main control |
|---|---|---|
| Hallucinations | Confident output that is false, such as invented facts, sources or software packages | Human review, checking against trusted sources, citations |
| Bias | Unfair or skewed results caused by unbalanced training data or design | Diverse, reviewed training data, testing results across groups |
| Explainability | Difficulty showing why a model reached a decision, which makes errors hard to find and decisions hard to defend | Prefer explainable models for important decisions, keep records, human oversight |
A hallucinated package name is a security risk as well as an accuracy problem. If a developer installs a package the model invented, an attacker may have registered that name with malicious code.
Risks to data and privacy
| Threat | What it is | Main control |
|---|---|---|
| Data loss | Sensitive data leaving the organization through prompts, uploads or model output | Approved AI tools only, data loss prevention (DLP), clear acceptable use policy |
| Privacy | Personal data used in training or prompts, stored, or revealed to other users | Data minimization, masking, privacy review, contracts with the AI provider |
A common case is an employee pasting customer records or source code into a public AI chat service. Once the data leaves, the organization may not control how it is stored or used.
Risks to sessions and systems
| Threat | What it is | Main control |
|---|---|---|
| Session hijacking | Stealing the session or token of a user's AI service, exposing conversation history and connected accounts | Short lived tokens, MFA, protecting tokens like any other session |
| Code execution | The model writes or runs code, or its output is passed to a system that runs it, so a malicious prompt leads to commands on the host | Run generated code in a sandbox, validate output before use, least privilege for AI tools and agents |
The more an AI system is allowed to do, such as run code, send email or change records, the more damage a successful injection can cause. Give AI agents only the permissions they need and require human approval for risky actions.
Ethical considerations
Ethical considerations cover fairness, transparency, accountability and the effect of AI decisions on people. An organization should know who is responsible for an AI decision, tell people when AI is used, and keep a human in the loop for decisions that affect people's rights or safety.
A governance policy for AI use ties these together. It lists approved tools, what data may be used with them and who reviews their output.
What to remember for the exam
- Poisoning corrupts training data. Evasion fools a model at use time with crafted input.
- Model manipulation tampers with the model itself.
- Prompt injection overrides instructions, directly or through hidden content. Jailbreaking gets around safety rules.
- Hallucinations are confident false output, so verify before trusting.
- Bias comes from skewed data. Explainability is about showing why a decision was made.
- Data loss and privacy risks come from what users put into AI tools.
- Session hijacking steals access to an AI service. Code execution turns model output into commands.
- Controls: trusted data, input and output filtering, least privilege, sandboxing, DLP and human review.
If hidden text in a document makes an AI assistant do something the user did not ask for, the answer is prompt injection.
Common mistakes
Confusing poisoning with evasion. Poisoning attacks training data. Evasion attacks the input when the model is used.
Confusing prompt injection with jailbreaking. Injection hijacks the instructions of an application. Jailbreaking aims to break the model's safety rules.
Trusting AI output as fact. A hallucination looks exactly as confident as a correct answer.
Giving AI agents broad permissions. An injected prompt can use every right the agent has.
Check yourself
What is data poisoning?
Adding false or malicious data to a model's training data so it learns the wrong behavior.
An AI assistant reads a web page that contains hidden instructions and then sends the user's files elsewhere. Which threat is this?
Indirect prompt injection.
What is the main control against code execution through an AI system?
Run generated code in a sandbox, validate output before use and give the AI tool least privilege.
Topic 10 of 10 in Threats, Vulnerabilities, and Attacks
Test yourself on Threats, Vulnerabilities, and Attacks
The full CompTIA Security+ SY0-801 question bank has 998 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by CompTIA. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.