Governance, Risk, and Compliance · Topic 7 of 7
AI Security Challenges
Understand AI risks including prompt injection, data poisoning, model theft, data leakage, hallucination and shadow AI, plus governance and guardrails.
- 5 min
- Includes a diagram
- Data poisoning attacks the training data, while prompt injection attacks the running model through input.
- Indirect prompt injection hides instructions in content the model retrieves.
- Treat model output as untrusted input before using it in other systems.
What it is
Artificial intelligence (AI) systems, especially large language models (LLMs), bring new attack paths on top of normal application risk. They learn from data, they accept free text as input, and their output can be wrong yet convincing.
The exam asks you to name the risk in a scenario and pick a proportionate control. It also expects you to recognize that AI can help attackers as well as defenders.
How it works
Risks to the model and its data
| Risk | What happens | Typical response |
|---|---|---|
| Data poisoning | An attacker corrupts training or fine tuning data so the model misbehaves | Vet data sources, track provenance, test for drift |
| Model theft | The model or its weights are copied or reconstructed by repeated queries | Access control, rate limits, monitoring |
| Model inversion and membership inference | An attacker infers training data or whether a person was in it | Limit output detail, privacy techniques, minimize sensitive training data |
| Evasion (adversarial input) | Crafted inputs make a model misclassify | Robust training, input checks |
| Supply chain compromise | A tampered pretrained model or dataset is used | Trusted sources, integrity checks, scanning |
Risks at the prompt and output
Prompt injection places instructions in input to override the system's intended behavior. In direct injection the user types it. In indirect injection the instructions hide in content the model reads, such as a web page, email or document.
Jailbreaking tries to bypass safety rules. Improper output handling (also called insecure output handling) occurs when an application trusts model output and passes it to a browser, shell or database, which can lead to classic injection flaws. Excessive agency means a model connected to tools, such as email or payment functions, can take harmful actions with too little oversight.
Risks to data and the organization
Employees may paste confidential data into a public AI service, which can expose it outside the organization. Using AI tools that the security team has not approved is called shadow AI. Retrieval augmented generation (RAG) connects a model to internal documents, so if access control is not enforced at retrieval time, users may see content they should not.
Other organizational risks include hallucination (confident but false output), biased results, legal questions over training data and intellectual property, and privacy duties for personal data used in training. Over-reliance without review can put wrong decisions into production.
AI used by attackers
Attackers use AI to write convincing phishing in many languages, produce deepfake audio and video for fraud, and automate reconnaissance and malware variation. Defenders should update awareness training, and add out of band verification for high value requests.
Controls and governance
- Governance: an acceptable use policy, an inventory of AI systems, owners, and risk assessment before adoption.
- Data controls: classify what may go to a model, use data loss prevention (DLP) at the boundary, and enforce document permissions in RAG.
- Technical guardrails: input and output filtering, least privilege for any tools the model can call, and sandboxing.
- Human in the loop: require review before high impact actions.
- Monitoring and testing: log prompts and outputs where lawful, and red team the system before and after release.
- Contracts: check whether a vendor trains on your data and how long it keeps it.
Useful references include the Open Worldwide Application Security Project (OWASP) Top 10 for LLM Applications, the National Institute of Standards and Technology (NIST) AI Risk Management Framework, and MITRE ATLAS (Adversarial Threat Landscape for Artificial Intelligence Systems). Some jurisdictions also have AI specific laws, which adds to compliance duties.
What to remember for the exam
- Data poisoning attacks the training data, while prompt injection attacks the running model through input.
- Indirect prompt injection hides instructions in content the model retrieves.
- Treat model output as untrusted input before using it in other systems.
- Shadow AI and pasting sensitive data into public tools are data leakage risks, handled by policy, approved tools and DLP.
- Limit what actions a model can take, because excessive agency is dangerous.
- RAG must respect the user's existing permissions.
- Model theft and inversion are addressed with access control, rate limiting and minimizing sensitive training data.
- Human review is a key safeguard for high impact AI decisions and actions.
When a scenario describes an assistant that can send email or run commands, the best answer usually limits its privileges and adds approval for risky actions.
Common mistakes
Calling a hallucination an attack. It is a reliability flaw, though attackers can exploit trust in wrong output.
Using one input filter as the only defense. Filters can be bypassed, so layer least privilege, output handling and monitoring.
Confusing poisoning with injection. Poisoning changes what the model learns. Injection changes what it does for one request.
Assuming a private model is safe. Internal models still need access control, logging and data governance.
Do not assume a vendor protects your prompts by default. Verify retention and training terms in the contract.
Check yourself
Work through these three questions.
What is indirect prompt injection?
Malicious instructions placed in content that a model later reads, such as a web page or document, rather than typed by the user.
Which control reduces harm from an AI agent with email access?
Apply least privilege to its tools and require human approval for high impact actions.
How does data poisoning differ from model inversion?
Poisoning corrupts the training data to change model behavior, while inversion extracts information about the training data from the model.
Topic 7 of 7 in Governance, Risk, and Compliance
Test yourself on Governance, Risk, and Compliance
The full CompTIA SecurityX CAS-005 question bank has 654 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by CompTIA. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.