Securing AI Systems · Topic 7 of 7
Attacks on LLM Applications and Compensating Controls
Analyze prompt injection, jailbreaking, insecure output handling, model DoS, plug-in flaws, excessive agency and overreliance, and pick the controls.
- 6 min
- Direct injection comes from the user; indirect injection hides in content the model reads.
- Jailbreaking removes safety limits; circumventing guardrails disguises a request to slip past filters.
- Insecure output handling is the application trusting model output; treat all output as untrusted.
What it is
Artificial intelligence (AI) applications built on large language models (LLMs) are attacked mainly through what goes into the model, what comes out, and what the model is allowed to do. These attacks need no access to training data, only a text box, a file or a web page the model reads.
Most of these attack names come from the Open Worldwide Application Security Project (OWASP) Top 10 for LLM applications. The objectives use the older names, such as insecure output handling and model denial of service (DoS), and you should recognise both those and the newer 2025 wording.
How it works
Input attacks
| Attack | What happens | Typical evidence |
|---|---|---|
| Prompt injection, direct | A user writes instructions that override the system prompt | Prompts telling the model to ignore its rules or reveal its instructions |
| Prompt injection, indirect | Instructions are hidden in content the model reads, such as a web page, email, document or retrieved chunk | The model acts on text the user never typed; the trigger is traced to a retrieved file |
| Jailbreaking | Role play, hypotheticals or layered requests persuade the model to drop its safety behaviour | Long, staged conversations ending in content the policy forbids |
| Circumventing AI guardrails | Encoding, misspelling, other languages or splitting a request across turns to slip past filters | Blocked attempts followed by an encoded or reworded variant that passes |
| Input manipulation | Crafted inputs designed to make the model misjudge or misclassify | Small, odd changes to input that flip the result |
Prompt injection is about hijacking instructions; jailbreaking is about removing safety limits. OWASP treats jailbreaking as a form of prompt injection, and many real attacks combine the two.
Output and integration attacks
| Attack | What happens | Typical evidence |
|---|---|---|
| Insecure output handling | The application trusts model output and passes it unchecked to a browser, shell, database or other system (2025 name: improper output handling) | Script in a chat reply runs in the user's browser; generated query runs against the database |
| Sensitive information disclosure | The model reveals personal data, secrets, the system prompt or confidential documents | Responses containing internal data; data loss alerts on chat output |
| Manipulating application integrations | The attacker abuses the connections between the model and other apps, such as email, calendars or ticketing | Unexpected emails, tickets or file shares created by the assistant |
| Insecure plug-in design | Plug-ins or tools accept free-form input, skip authorisation or run with excessive rights | A plug-in performs actions for one user with another user's access |
| Excessive agency | The model or agent has more functions, permissions or autonomy than its task needs | An assistant meant to read mail can also delete it or send it externally |
Availability and trust problems
| Attack or risk | What happens | Typical evidence |
|---|---|---|
| Model DoS | Huge, complex or recursive requests exhaust compute, context or budget (2025 name: unbounded consumption) | Latency spikes, very long prompts, token use and cost jump |
| Overreliance | People or systems trust model output without checking it | Hallucinated fixes, invented packages or wrong findings acted on without review |
Overreliance is not an attack in itself, but attackers exploit it. If staff install any package the assistant suggests, an attacker can publish a malicious package under a name the model tends to invent.
Evidence from a gateway log
09:31:07 app=mail-assistant user=akhan action=summarise source=inbox/msg-4471
09:31:08 model_tool_call=send_email to=external-address@example.net attachments=3
09:31:08 policy=excessive_agency_check verdict=ALLOW reason=tool_in_scopeThe user asked only for a summary, yet the model called a send tool straight after reading one message. That points to indirect prompt injection, made dangerous by excessive agency.
Compensating controls
| Control | Best against |
|---|---|
| Prompt firewalls | Direct and indirect prompt injection, jailbreak phrasing, guardrail evasion, sensitive data in prompts |
| Model guardrails | Harmful or policy-breaking output, sensitive information disclosure |
| Prompt templates | Injection, by separating trusted instructions from untrusted input |
| Access controls | Disclosure of data the user should not see, plug-in misuse |
| Least privilege | Excessive agency, insecure plug-in design, manipulated integrations |
| Rate limiting | Model DoS, brute-force jailbreak attempts, extraction |
| Data integrity controls | Poisoned retrieval sources that carry indirect injection |
| Encryption | Disclosure of prompts, responses and logs in transit or at rest |
Two further controls appear in nearly every answer: treat output as untrusted (encode and validate it, never execute it directly) and require human approval for high-impact actions.
A worked example
A help desk chatbot starts replying with links that run script in agents' browsers. Investigation shows a customer submitted a ticket containing hidden instructions, and the chatbot's reply was inserted into the agent console without encoding.
Two weaknesses combined: indirect prompt injection and insecure output handling. The compensating controls are output encoding in the console, a prompt firewall on ticket text, and a template that marks ticket content as untrusted data.
How the exam asks about this
- "An AI assistant summarising web pages begins sending users' data to an outside address after visiting one site. Which of the following attacks is most likely?" Indirect prompt injection.
- "A chatbot's generated text is inserted directly into a web page, and script in it runs in visitors' browsers. Which of the following vulnerabilities does this show?" Insecure output handling.
- "An email agent that only needs to read and summarise messages can also send and delete them. Which of the following compensating controls best reduces the risk?" Least privilege, removing the send and delete tools.
What to remember for the exam
- Direct injection comes from the user; indirect injection hides in content the model reads.
- Jailbreaking removes safety limits; circumventing guardrails disguises a request to slip past filters.
- Insecure output handling is the application trusting model output; treat all output as untrusted.
- Excessive agency and insecure plug-in design are fixed with least privilege, authorisation and approval steps.
- Model DoS is countered by rate limits and token limits; overreliance by human validation.
No prompt firewall or guardrail stops every injection. Design so that a successful injection can do little harm: least privilege, output encoding and human approval for anything irreversible.
Common mistakes
Calling every manipulation a jailbreak. If the goal is to make the model follow attacker instructions or leak data, it is prompt injection. If the goal is forbidden content, it is jailbreaking.
Fixing excessive agency with a better system prompt. Telling the model not to misuse a tool does not remove the tool. Remove the permission.
Overlooking indirect sources. Retrieved documents, emails and web pages carry injected text too. Inspect and label them, not just what the user types.
Check yourself
What is the difference between direct and indirect prompt injection?
Direct injection is typed by the user into the prompt, while indirect injection is hidden in content the model reads, such as a web page or document.
Which compensating control best addresses excessive agency?
Least privilege, giving the model or agent only the tools and permissions its task needs, with human approval for high-impact actions.
What is overreliance?
Trusting model output without verification, so hallucinated or manipulated results are acted on.
Topic 7 of 7 in Securing AI Systems
Test yourself on Securing AI Systems
The full CompTIA SecAI+ CY0-001 question bank has 654 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by CompTIA. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.