Attacks and Exploits · Topic 6 of 10
Attacks on AI Systems
Understand prompt injection, data poisoning, model extraction, adversarial inputs and other attacks on AI and LLM systems, with defences and frameworks.
- 5 min
- Prompt injection manipulates a model through text, either directly or indirectly through content it reads.
- Jailbreaking targets safety rules, and system prompt leakage exposes hidden instructions.
- Data poisoning corrupts training data, and it can plant hidden behaviour.
What it is
Artificial intelligence (AI) systems, especially those built on large language models (LLMs), have a new attack surface. They accept natural language, depend on training data, and often have access to tools and company data.
The exam covers this at a concept level: recognise the attack from a description, know what it damages, and choose a defence. This page does not give attack strings, because the ideas are what the exam tests.
How it works
Each stage in the diagram can be attacked: the data that trains the model, the model itself, and the application wrapped around it.
Prompt injection
An LLM cannot reliably tell instructions from data, because both arrive as text. Prompt injection is crafting input that makes the model follow the attacker's instructions instead of the developer's.
| Type | Description |
|---|---|
| Direct prompt injection | The attacker types the malicious instructions into the chat or input field |
| Indirect prompt injection | Instructions are hidden in content the model reads later, such as a web page, email or document |
| Jailbreaking | Persuading a model to ignore its safety rules |
| System prompt leakage | Getting the model to reveal its hidden instructions |
The danger rises when the model has tools. A model that can send email, read files or call application programming interfaces (APIs) can be steered into doing real harm, and giving it more functions, permissions or autonomy than its task needs is called excessive agency.
Model and data attacks
| Attack | What happens | Harm |
|---|---|---|
| Data poisoning | Bad or biased samples are added to training or fine-tuning data | The model behaves wrongly or contains a hidden trigger |
| Model inversion | Outputs are used to reconstruct training data | Privacy loss |
| Membership inference | Testing whether a record was in the training set | Privacy loss |
| Model extraction (theft) | Many queries are used to copy the model's behaviour | Loss of intellectual property |
| Adversarial examples (evasion) | Inputs slightly altered to fool a classifier | Wrong decisions, for example a missed malware detection |
| Training data extraction | Prompting the model to repeat memorised content | Leak of secrets or personal data |
| Supply chain compromise | Malicious pre-trained models, libraries or datasets | Backdoors in the system |
Application-level weaknesses
- Insecure output handling: the application trusts the model's output and passes it to a browser, database or shell, which can lead to cross-site scripting (XSS) or injection.
- Sensitive information disclosure: the model returns confidential data from its context or from connected stores.
- Excessive agency: the AI agent has more tools, permissions or autonomy than its task needs, so a manipulated model can act with real impact.
- Denial of service by resource use: very long or costly prompts drain capacity or budget.
- Weak access control around retrieval: a document search feature returns files that the asking user should not see.
Defences
Defences work in layers, because no single filter solves the language problem.
| Control | Purpose |
|---|---|
| Least privilege for the model's tools and data | Limits damage if the model is manipulated |
| Human approval for sensitive actions | Stops silent harmful actions |
| Treat model output as untrusted | Encode or validate it before use |
| Input and output filtering | Catches known bad patterns, but not all |
| Separate trusted instructions from untrusted content | Reduces indirect injection |
| Data provenance and review | Reduces poisoning and supply chain risk |
| Rate limiting and monitoring | Slows extraction and abuse |
| Access control on retrieved data | Enforces who can see which documents |
| Red team testing | Finds weaknesses before attackers do |
Frameworks
- OWASP Top 10 for LLM Applications: the leading risk list for LLM-based systems.
- MITRE ATLAS: a knowledge base of adversary tactics against AI systems, modelled on MITRE ATT&CK.
- NIST AI Risk Management Framework: guidance on managing AI risk.
When the exam describes hidden instructions inside a web page or email that an AI assistant summarises, the answer is indirect prompt injection.
What to remember for the exam
- Prompt injection manipulates a model through text, either directly or indirectly through content it reads.
- Jailbreaking targets safety rules, and system prompt leakage exposes hidden instructions.
- Data poisoning corrupts training data, and it can plant hidden behaviour.
- Model extraction copies a model through queries, and model inversion and membership inference threaten privacy.
- Adversarial examples are small input changes that make a model decide wrongly.
- Excessive agency and insecure output handling make injection more dangerous.
- Defend with least privilege, human approval, untrusted-output handling, monitoring and layered filters.
- MITRE ATLAS and the OWASP Top 10 for LLM Applications are the framework names to know.
Common mistakes
Believing a filter solves prompt injection. Filters reduce known patterns but cannot fully separate instructions from data.
Trusting the model's output. If it feeds a browser or command, it must be encoded and validated like any user input.
Confusing poisoning with evasion. Poisoning corrupts the training stage, and evasion tricks a trained model at use time.
Ignoring the tools around the model. The permissions of connected tools often decide how serious an injection is.
Overlooking data privacy. Models trained on sensitive data can leak it.
Check yourself
What is indirect prompt injection?
Hidden instructions placed in content that the model later reads, such as a web page or email.
Which attack corrupts the data used to train a model?
Data poisoning.
Name one control that limits the damage of a successful injection.
Least privilege for the model's tools, or human approval for sensitive actions.
Topic 6 of 10 in Attacks and Exploits
Test yourself on Attacks and Exploits
The full CompTIA PenTest+ PT0-003 question bank has 653 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by CompTIA. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.