AI Governance and Program Management · Topic 5 of 6
AI Security Program Development and Management
Building and running the AI security program: gap analysis, roadmap, resources, AI security metrics, choosing AI security tools and role-based training.
- 5 min
- Extend the existing security program; do not build a parallel one.
- Start with a gap analysis against policy, regulation and a framework.
- KPIs measure program performance; KRIs warn of rising risk.
What it is
The AI security program is the set of people, processes, technologies and activities that put the AI strategy and policy into practice. It is not a separate empire: it extends the existing information security program so that AI systems get the same disciplined treatment as other assets, plus the controls that AI specifically needs.
Managing the program means measuring whether it works, choosing tools that fit, building skills and reporting to management in business terms.
How it works
Developing the program
| Step | What happens |
|---|---|
| Define scope | All AI systems in the inventory, including vendor and embedded AI |
| Assess current state | Gap analysis against policy, regulation and a chosen framework such as ISO/IEC 42001 or the NIST AI RMF |
| Define target state | From the AI security strategy and risk appetite |
| Build the roadmap | Prioritized initiatives, owners, budget and milestones |
| Integrate | Add AI to existing processes: risk assessment, change management, vendor management, secure development, incident response |
| Operate and improve | Monitor metrics, review and update |
Integration is the key design choice. Adding AI checks to the change advisory process, the vendor questionnaire and the secure development life cycle is cheaper and more durable than building a parallel process.
Resources and skills
AI security needs skills that many security teams lack: understanding how models are trained and attacked, evaluating model behaviour and reading model documentation. Options include training existing staff, hiring specialists, partnering with data science teams and using external testers for red teaming. A small central team often sets standards while champions in AI development teams apply them.
AI security metrics
Metrics must answer management's questions: is AI risk within appetite, and is the program improving? Good metrics are tied to objectives, measurable, repeatable and reported with a target.
| Metric type | Examples |
|---|---|
| Coverage (KPI) | Percentage of inventoried AI systems with a current risk assessment; percentage of high-risk systems with human oversight defined |
| Program performance (KPI) | Time from use case request to approval; red team findings closed within target |
| Risk indicators (KRI) | Number of unapproved AI tools detected; rising rate of prompt injection attempts; model drift beyond threshold |
| Operational | Guardrail block rates, AI incidents by severity, mean time to contain AI incidents |
| Awareness | Training completion by role; data loss events involving AI tools |
A key risk indicator signals that risk is increasing and may need action; a key performance indicator shows whether the program is meeting its goals. Reports to the board use a few trend metrics, not raw technical counts.
Reviewing and implementing AI security tools
Tool categories that appear in AI security programs include:
- AI discovery and posture management: finds AI services, models and data stores and checks their configuration.
- Model scanning: checks model files and dependencies for malicious code and known weaknesses before use.
- Input and output guardrails: filter prompts and responses for injection attempts, sensitive data and harmful content.
- Data loss prevention for AI: detects sensitive data sent to AI tools.
- AI red teaming and evaluation tools: automate adversarial and safety testing.
- Monitoring: feeds AI logs and alerts into the existing security monitoring platform.
Tools are selected against requirements derived from risk, not vendor claims. The manager checks integration with existing tools, the vendor's own data handling, false positive rates and the effort to operate the tool.
AI security awareness training
| Audience | Focus |
|---|---|
| All staff | Acceptable use, what data not to share, checking AI output, AI-enabled phishing and deepfakes, reporting |
| Developers and data scientists | Secure AI development, common LLM application risks, data handling, testing |
| Business owners | Their accountability, risk tiers, oversight duties |
| Executives and board | AI risk, regulatory exposure, appetite decisions |
| Procurement | AI questions to ask vendors and contract terms |
Under the EU AI Act, providers and deployers must take measures to support the AI literacy of staff who deal with AI systems, a duty that has applied since 2 February 2025, which makes role-based training an obligation as well as good practice.
A worked example
The CISO asks whether the AI security program is working. The team reports 40 detailed technical counts each month, and management cannot tell whether risk is improving.
The security manager replaces them with five metrics tied to objectives: assessment coverage of high-risk AI, overdue red team findings, unapproved AI tools detected, AI incidents and training completion. Each has a target and a trend line, and the report explains what action is needed when a target is missed.
How the exam asks about this
- "What is the BEST way to build an AI security program when an information security program already exists?" Integrate AI requirements into existing processes and governance.
- "Which metric BEST shows management whether AI risk is increasing?" A key risk indicator with a threshold, such as unapproved AI tools detected over time.
- "What should drive the selection of an AI guardrail tool?" Requirements derived from the organization's AI risk assessment.
What to remember for the exam
- Extend the existing security program; do not build a parallel one.
- Start with a gap analysis against policy, regulation and a framework.
- KPIs measure program performance; KRIs warn of rising risk.
- Choose tools from risk-based requirements, and check how they handle your data.
- Training is role-based, and the EU AI Act requires measures to support AI literacy.
Metrics that management cannot act on are noise. Each metric needs an owner, a target and a defined action when it crosses a threshold.
Common mistakes
Measuring activity instead of outcome. The number of prompts filtered says little on its own; the share of high-risk AI systems assessed and protected tells management more.
Buying tools before requirements. A tool chosen without a risk assessment may protect the wrong thing or duplicate an existing control.
One training course for everyone. Developers, executives and general staff face different AI risks and need different content.
Check yourself
What is the difference between a KPI and a KRI?
A KPI shows whether the program is meeting its goals; a KRI signals that a risk is increasing and may need action.
Why integrate AI security into existing processes?
It is cheaper and more durable, and it ensures AI is covered wherever change, procurement and development already happen.
What should general staff AI awareness training cover?
Acceptable use, what data not to share, checking AI output, AI-enabled phishing and deepfakes, and how to report problems.
Topic 5 of 6 in AI Governance and Program Management
Test yourself on AI Governance and Program Management
The full ISACA AAISM question bank has 747 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by ISACA. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.