AI Risk Management · Topic 3 of 4
AI Testing, Red Teaming and Vulnerability Management
Designing AI security testing: robustness and safety evaluation, red teaming rules, model and dependency scanning, AI vulnerability tracking and retesting.
- 5 min
- AI testing combines performance, robustness, adversarial, safety, bias and conventional security tests.
- Red teaming needs scope, approved rules of engagement and safe handling of outputs.
- Many AI weaknesses have no patch; limit impact with layered controls.
What it is
AI vulnerability management finds weaknesses in AI systems, rates them by risk, gets them treated and confirms the fix. It covers the familiar parts of an AI application, such as servers, libraries and APIs, and the AI-specific parts: how the model behaves under hostile or unusual input, how it handles data and what it is allowed to do.
Testing is how weaknesses are found. For AI that includes red teaming: structured, adversarial testing by people and tools that try to make the system fail in the ways a real attacker or misuse would.
How it works
Types of AI testing
| Test | Purpose |
|---|---|
| Performance validation | Accuracy and error rates on independent test data against thresholds |
| Robustness testing | Behaviour with noisy, unusual or out-of-distribution inputs |
| Adversarial testing | Resistance to evasion, prompt injection and jailbreak attempts |
| Safety and content evaluation | Rate of harmful, biased or policy-violating outputs |
| Bias and fairness testing | Differences in outcomes between groups |
| Data leakage testing | Whether the system reveals training data, system prompts or other users' data |
| Agent and tool testing | Whether the AI can be pushed into unauthorized actions |
| Application security testing | Conventional penetration testing, code review and API testing of the surrounding application |
| Model and dependency scanning | Malicious code in model files, vulnerable libraries, unsafe serialization formats |
No single test is enough. Automated evaluation gives breadth and repeatability; human red teamers find creative failures that automated suites miss.
Red teaming done properly
Red teaming is planned like any other security test:
- Scope and objectives tied to the risk assessment: which harms matter most for this use case.
- Rules of engagement approved by the business owner, covering systems, data, timing and limits.
- Legal and ethical review, especially when testing for harmful content.
- Diverse testers: security specialists, domain experts and people who understand the user population.
- Safe handling of test outputs, which may themselves be harmful or sensitive.
- Reporting of findings with severity, reproduction conditions and recommended treatment.
Red teaming is repeated at major changes: a new model version, new tools for an agent, new data sources or a new user group.
Why AI vulnerabilities are different
Many AI weaknesses cannot be patched in the usual sense. A model's tendency to follow injected instructions comes from how it works, so there is no single fix. The response is layered: reduce what the model can do, filter inputs and outputs, separate trusted instructions from untrusted content, require human approval for high-impact actions and monitor for abuse.
Findings may also be probabilistic. An attack that works one time in fifty is still a finding; severity reflects likelihood and impact across many attempts.
Vulnerability sources and tracking
| Source | Example |
|---|---|
| Internal testing and red teaming | Findings from evaluations before release |
| Advisories | Published vulnerabilities in AI frameworks, libraries and model serving software |
| Vendors | Model provider security notices and changes |
| Threat intelligence | New techniques catalogued in MITRE ATLAS or reported by researchers |
| Responsible disclosure and bug bounty | External researchers reporting AI flaws |
| Monitoring | Production patterns that suggest a weakness is being exploited |
AI findings go into the same vulnerability register as other findings, with an owner, risk rating, due date and status, so they are reported and escalated consistently. Linking findings to the AI inventory and AI BOM shows every system affected by a shared weakness.
Treatment and retesting
| Treatment | Example |
|---|---|
| Fix the component | Update a vulnerable library; replace an unsafe model file |
| Change the model | Retrain or fine-tune; move to a better-performing model |
| Change the architecture | Remove a tool from an agent; add permission-aware retrieval |
| Add compensating controls | Input and output filtering, rate limits, human approval |
| Accept | Business owner accepts residual risk within appetite |
Every fix is retested, and the test is added to the regression suite so the weakness does not quietly return in a later model version.
A worked example
Before launch, red teamers show that a support assistant can be talked into revealing its system prompt, which contains an internal API key. They also find that an older library in the serving stack has a published vulnerability.
The manager rates the key exposure as high: secrets do not belong in prompts, so the key is removed, rotated and stored in a secrets manager. The library is updated. Both tests join the regression suite, and launch proceeds after a clean retest.
How the exam asks about this
- "What should be defined FIRST before red teaming an AI system?" Scope, objectives and rules of engagement, approved by the business owner.
- "Prompt injection cannot be fully fixed. What is the BEST approach?" Layered controls that limit impact, such as least privilege, filtering and human approval.
- "When should an AI system be retested?" After fixes and after significant changes such as a new model version or new tools.
What to remember for the exam
- AI testing combines performance, robustness, adversarial, safety, bias and conventional security tests.
- Red teaming needs scope, approved rules of engagement and safe handling of outputs.
- Many AI weaknesses have no patch; limit impact with layered controls.
- Track AI findings in the normal vulnerability register, linked to the inventory.
- Retest after fixes and changes, and keep tests as regression checks.
Passing a red team exercise once does not prove a model is safe. Model updates, new tools and new attack techniques can reopen weaknesses, so testing must be repeated.
Common mistakes
Testing only the model. Many AI incidents come from the surrounding application, such as unvalidated output or excessive permissions.
Dismissing low-rate findings. An attack that succeeds occasionally can still be exploited at scale.
Keeping AI findings in a separate spreadsheet. They then miss the normal reporting, escalation and due-date tracking.
Check yourself
Why are many AI vulnerabilities treated with layered controls instead of a patch?
Some weaknesses, such as susceptibility to prompt injection, come from how models work, so impact is reduced with several controls rather than one fix.
What belongs in red team rules of engagement?
The systems and data in scope, timing, limits, approvals and how findings and harmful outputs are handled.
Why add each fixed AI finding to a regression suite?
So later model versions or changes are checked and the weakness does not return unnoticed.
Topic 3 of 4 in AI Risk Management
Test yourself on AI Risk Management
The full ISACA AAISM question bank has 747 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by ISACA. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.