AI Operations · Topic 7 of 8
AI Threat, Vulnerability and Access Programs
Evaluating AI threat and vulnerability management and identity and access management: threat models, scanning, AI bill of materials, agent and API access.
- 5 min
- Confirm AI assets are in the vulnerability program's scope.
- Threat model each significant AI system with OWASP or ATLAS.
- An AI bill of materials speeds response to component issues.
What it is
Knowing the threats is not enough. The organization needs programs that find, assess and fix AI weaknesses continuously, and that control who and what can reach AI systems. The outline asks the auditor to evaluate the threat and vulnerability management program and the identity and access management program specific to AI.
The auditor's focus is the same as for any program: is it designed to cover AI assets, is it operating, and are findings fixed on time?
How it works
Threat and vulnerability management for AI
| Program element | What the auditor expects |
|---|---|
| Scope | AI models, datasets, pipelines, retrieval stores, agents, APIs and AI libraries are in scope |
| Threat modelling | Done for each significant AI system, using references such as OWASP or MITRE ATLAS |
| Intelligence | Feeds include AI-specific vulnerabilities and attack trends |
| Testing | Scanning of AI libraries and containers, plus AI-specific testing and red teaming for high-risk systems |
| Rating | Severity reflects AI impact, such as harm to people or data leakage |
| Remediation | Owners, deadlines by severity, tracking to closure |
| Exceptions | Risk acceptance by the business owner, with expiry |
The commonest gap is scope: the existing vulnerability program covers servers and applications but not models, model files or retrieval stores.
AI bill of materials
An AI bill of materials lists the components of an AI system: base models and their versions, datasets, libraries and services. Like a software bill of materials, it lets the organization find affected systems quickly when a component is found vulnerable or compromised. The auditor checks it exists, is current and is used in vulnerability response.
Model and component provenance
Pretrained models and datasets downloaded from public sources can carry hidden behaviour or malicious code in model files. Good practice is to use approved sources, verify integrity with hashes or signatures, scan model files and test downloaded models before use. The auditor samples models and checks provenance records.
Vendor and supply chain links
For vendor AI services, the organization relies on the vendor's own vulnerability management. The auditor checks contract terms on vulnerability disclosure and patching, reviews independent assurance reports, and confirms the organization tracks vendor security notices.
Identity and access management for AI
AI systems create new identities and new access paths.
| Access area | Control the auditor tests |
|---|---|
| Training data and feature stores | Least privilege; access reviews |
| Model registry and pipelines | Only authorised people can approve or deploy |
| Model weights and files | Protected as high-value assets |
| Prompts and configuration | Edit rights restricted, changes logged |
| AI APIs and keys | Keys stored securely, rotated, scoped, rate limited |
| Agent and service identities | Each agent has its own identity with minimum permissions |
| Retrieval access | The AI shows users only what they are already entitled to see |
Non-human identities for agents and services are a growing risk. The auditor checks they are inventoried, owned, reviewed like user accounts and not shared.
Permission-aware retrieval matters for enterprise assistants. If the assistant searches all documents with its own broad access, any user can retrieve content they should not see. The auditor tests by comparing what a sample user can retrieve through the AI with their normal entitlements.
Access reviews and logging
Periodic access reviews should include AI roles: who can train, approve, deploy and edit prompts, and which agents hold which permissions. Access to models and AI APIs should be logged and monitored for unusual query volumes, which can signal extraction or abuse.
A worked example
An auditor examines an internal knowledge assistant. It uses one service account with read access to the entire document store. A test user from marketing asks about salary bands and receives a summary of an HR document they cannot open directly.
The finding is that retrieval does not enforce user entitlements and the service identity has excessive access. The recommendation is to enforce the user's own permissions in retrieval and reduce the service account's scope.
How the exam asks about this
- "What is the auditor's GREATEST concern if the vulnerability management program excludes AI models and datasets?" AI-specific weaknesses are not identified or remediated.
- "Which control BEST prevents an enterprise AI assistant from disclosing documents to unauthorised users?" Retrieval that enforces each user's existing access rights.
- "What is the BEST way to respond quickly when a widely used open source model is found to be compromised?" Use a current AI bill of materials to identify affected systems.
What to remember for the exam
- Confirm AI assets are in the vulnerability program's scope.
- Threat model each significant AI system with OWASP or ATLAS.
- An AI bill of materials speeds response to component issues.
- Agents and services need their own least-privilege identities.
- Retrieval must respect user entitlements.
An AI assistant with broad service access can become a way around every access control in the organization. Test what users can see through the AI, not just what they can open directly.
Common mistakes
Assuming the existing scanner covers AI. Model files, retrieval stores and prompts are often outside its scope.
Sharing one key across AI applications. It removes accountability and widens the impact of a leak.
Skipping provenance for downloaded models. Public models can carry hidden behaviour or malicious code.
Check yourself
What is an AI bill of materials?
A list of an AI system's components, such as base models, datasets, libraries and services, used to find affected systems when a component has a problem.
Why do AI agents need their own identities?
So their permissions can be limited, reviewed and logged, and actions traced to a specific agent rather than a shared account.
How can an auditor test permission-aware retrieval?
By comparing what a sample user can retrieve through the AI with what that user is entitled to open directly.
Topic 7 of 8 in AI Operations
Test yourself on AI Operations
The full ISACA AAIA question bank has 750 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by ISACA. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.