AI Technologies and Controls · Topic 7 of 7
AI Security Monitoring, Metrics and Tools
Monitoring AI in production: what to log, detection use cases, drift and integrity monitoring, security metrics and thresholds, and reviewing AI tools.
- 5 min
- Log identity, prompts, outputs, retrieved sources, tool actions, model version, guardrail decisions and cost.
- Prompt and output logs are sensitive and need their own protection.
- Send AI logs to the central monitoring platform for correlation.
What it is
AI systems change after deployment. Inputs shift, vendors update models, attackers probe defences and controls drift out of tune. Monitoring watches AI systems in production so problems are detected early, and metrics tell management whether AI security is working.
Monitoring AI combines classic security monitoring, such as authentication and access events, with AI-specific signals: what users ask, what the model answers, what tools it calls and whether its behaviour is changing.
How it works
What to log
| Data | Why |
|---|---|
| User and service identity | Attribution and abuse detection |
| Prompts and inputs | Detect injection, jailbreaks and data leakage attempts |
| Outputs | Detect sensitive data, harmful content, unusual answers |
| Retrieved sources | Show what content shaped an answer |
| Tool and agent actions | Attribute and review every action taken |
| Model and configuration version | Link behaviour to a specific version |
| Guardrail decisions | Measure blocks, bypasses and false positives |
| Usage and cost | Detect abuse and runaway consumption |
Logs of prompts and outputs contain sensitive data. They need access control, encryption, retention limits and privacy review, and may need masking of personal data.
Detection use cases
| Signal | Possible issue |
|---|---|
| Repeated injection or jailbreak patterns from one account | Active attack |
| Very high query volumes with systematic variations | Model extraction attempt |
| Sensitive data or secrets appearing in outputs | Data leakage or system prompt leakage |
| Agent calling tools it rarely uses, or at unusual times | Manipulated agent or compromised account |
| Sudden cost or token spikes | Unbounded consumption or abuse |
| Deployed model hash differs from the approved version | Tampering or unauthorized change |
| Unapproved AI services in network or cloud logs | Shadow AI |
AI logs are sent to the organization's security monitoring platform so they can be correlated with other events, such as a compromised account using the AI assistant.
Monitoring model behaviour
| Type | What it watches |
|---|---|
| Data drift | Inputs differ from those the model was trained on |
| Concept drift | The relationship between inputs and correct outputs has changed |
| Performance | Accuracy, error rates and quality scores against thresholds |
| Fairness | Outcome differences between groups |
| Safety | Rate of harmful or policy-violating outputs |
| Integrity | Deployed model, prompts and configuration match the approved versions |
Each has a threshold agreed in the risk assessment. Crossing it triggers a defined response: investigation, retraining, rollback or reassessment of risk.
Security metrics for AI
| Metric | What it tells management |
|---|---|
| Percentage of AI systems in the inventory with current risk assessments | Coverage of governance |
| Percentage of high-risk AI with monitoring and human oversight in place | Coverage of key controls |
| Guardrail block rate and false positive rate | Control effectiveness and usability |
| Number of AI incidents by severity, and mean time to contain | Operational resilience |
| Open red team and vulnerability findings past due date | Remediation discipline |
| Unapproved AI tools detected | Shadow AI trend |
| Drift and fairness threshold breaches | Model health |
Metrics are defined with an owner, data source, target and reporting frequency. Reporting shows trends against targets and the action taken, not raw counts.
Reviewing AI security tools
AI security tools, such as guardrails, AI gateways, model scanners, AI posture management and AI-aware DLP, are reviewed like any other part of the security program. The review asks whether each tool still addresses current risks, how it performs in the organization's environment, how it handles the organization's data, whether it integrates with monitoring and response, and whether overlapping tools can be consolidated. New tools are piloted against measured requirements before wider rollout.
A worked example
Monitoring shows a sharp rise in token usage on an internal assistant at night, from one service account, with prompts that vary slightly but ask for complete outputs on many topics.
The analyst raises an alert under the extraction and abuse use case. The account is suspended, and investigation finds its credentials were exposed in a code repository.
The credentials are rotated, a per-account rate limit is added, and the detection rule is added to the regression set. The incident appears in that month's metrics with time to contain.
How the exam asks about this
- "Where should AI system logs be sent to detect attacks BEST?" The organization's central security monitoring platform, for correlation with other events.
- "A production model's accuracy falls below its threshold. What should happen?" The defined response: investigate drift, then retrain, roll back or reassess risk as agreed.
- "Which metric BEST shows whether AI governance covers the organization's AI use?" The percentage of inventoried AI systems with current risk assessments.
What to remember for the exam
- Log identity, prompts, outputs, retrieved sources, tool actions, model version, guardrail decisions and cost.
- Prompt and output logs are sensitive and need their own protection.
- Send AI logs to the central monitoring platform for correlation.
- Monitor drift, performance, fairness, safety and integrity against agreed thresholds.
- Metrics need owners and targets; review AI security tools against current risks.
Monitoring without defined thresholds and responses only collects data. Each AI metric and signal needs a threshold and an agreed action when it is crossed.
Common mistakes
Logging only infrastructure events. Without prompts, outputs and tool actions, AI attacks are invisible.
Keeping AI logs in a separate silo. Attacks that span accounts, endpoints and AI systems are missed without correlation.
Watching accuracy but not fairness or safety. A model can stay accurate overall while becoming unfair to a group or unsafe in some outputs.
Check yourself
What is concept drift?
A change in the relationship between inputs and correct outputs, so a model trained on past patterns becomes less accurate.
Why must prompt and output logs be protected?
They often contain personal and confidential data, so they need access control, encryption and retention limits.
What signal may indicate a model extraction attempt?
Very high query volumes with systematic variations, often from one account, aimed at reproducing the model's behaviour.
Topic 7 of 7 in AI Technologies and Controls
Test yourself on AI Technologies and Controls
The full ISACA AAISM question bank has 747 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by ISACA. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.