Securing AI Systems · Topic 5 of 7
Monitoring and Auditing AI Systems
Monitor AI prompts, responses, logs, confidence, request rates and cost, and audit models for hallucinations, accuracy, bias, fairness and access.
- 5 min
- Prompt monitoring inspects both the query and the response.
- Sanitise logs before storing them, and protect them with access control, encryption and tamper-evident storage.
- Low response confidence should route output to a human or a fallback.
What it is
Monitoring watches an artificial intelligence (AI) system while it runs, so attacks, abuse, failures and cost spikes are spotted quickly. Auditing reviews it periodically against standards for quality and compliance.
AI systems need more than ordinary uptime checks. A model can be fully available and still be leaking data, answering wrongly or being steered by an attacker, and only monitoring of its prompts and responses shows that.
How it works
Prompt monitoring
Prompt monitoring inspects both sides of every exchange.
| Side | What to watch for |
|---|---|
| Query (prompt) | Injection phrasing such as "ignore previous instructions", encoded or obfuscated text, secrets or personal data pasted in, unusual length, repeated probing |
| Response | Sensitive data, system prompt text, harmful content, unexpected code or links, answers outside the system's purpose |
Repeated, slightly varied prompts from one user are a common sign of jailbreak attempts or model extraction. Alert on the pattern, not just single prompts.
Log monitoring, sanitization and protection
Logs are evidence, so they must be complete, safe to store and trustworthy.
- Log monitoring: feed AI gateway, model and agent logs into the security information and event management (SIEM) system and correlate them with identity and network events.
- Log sanitization: mask or redact secrets, credentials and personal data before logs are stored, so the logs do not become a second leak.
- Log protection: restrict access, encrypt logs, keep them in write-once or tamper-evident storage, and set retention to match policy.
A log excerpt
2026-10-19T09:14:02Z gw user=jlee app=helpdesk tokens_in=3810 tokens_out=12 verdict=BLOCK rule=injection_pattern
2026-10-19T09:14:09Z gw user=jlee app=helpdesk tokens_in=3795 tokens_out=11 verdict=BLOCK rule=injection_pattern
2026-10-19T09:14:15Z gw user=jlee app=helpdesk tokens_in=3802 tokens_out=640 verdict=ALLOW confidence=0.41Two blocked prompts of almost the same size are followed by a variant that passed and produced a long answer with low confidence. The analyst should review the allowed response for leaked data and treat the account as a likely jailbreak attempt.
Response confidence level
Many models and classifiers can report a confidence score for their output. Low confidence means the answer is more likely to be wrong.
Use it as a routing signal: high-confidence results can flow on, while low-confidence results go to a human or trigger a safe fallback answer. A sudden fall in average confidence can also signal drift or adversarial inputs.
Rate monitoring
Rate monitoring tracks requests and tokens per user, key and application over time. Spikes can mean scraping, model theft by mass querying, a runaway agent loop or a model denial of service (DoS) attempt.
AI cost monitoring
AI usage is usually billed per token and per unit of compute, so cost is an early warning of misuse.
| Cost element | What drives it |
|---|---|
| Prompts | Input tokens, including long context and retrieved documents |
| Storage | Vector stores, training data, model versions, logs |
| Response | Output tokens; long generated answers cost more |
| Processing | Graphics processing unit (GPU) time for training, fine-tuning and inference |
Set budgets and alerts per application and user. An unexpected bill is often the first sign of a leaked application programming interface (API) key or an agent stuck in a loop.
Auditing for quality and compliance
| Audit area | What it checks | Method |
|---|---|---|
| Hallucinations | The model inventing facts, sources or commands | Sample responses and check them against trusted sources |
| Accuracy | Correctness against expected results | Benchmark test sets, comparison with ground truth |
| Bias and fairness | Different outcomes for different groups | Compare results across demographic or other groups |
| Access | Who used, changed or exported the model and data | Review access logs, role assignments and approvals |
A worked example
A finance team's monthly AI bill doubles. Cost monitoring shows almost all new spend on one application key, and rate monitoring shows steady traffic all night when nobody works.
Log review finds the key was committed to a public code repository. The team revokes the key, issues a new one from the vault, adds a per-key budget alert and audits access to confirm what data the misuse retrieved.
How the exam asks about this
- "Investigators find customers' full card numbers stored in the AI gateway's prompt logs. Which of the following should be implemented?" Log sanitization that masks or redacts sensitive data before logs are written.
- "A chatbot's answers that score low confidence must be reviewed by a person before they are sent. Which of the following does this use?" Response confidence level monitoring with routing to human review.
- "An organisation must show regulators that its loan model treats applicant groups equally. Which of the following audits should it run?" A bias and fairness audit.
What to remember for the exam
- Prompt monitoring inspects both the query and the response.
- Sanitise logs before storing them, and protect them with access control, encryption and tamper-evident storage.
- Low response confidence should route output to a human or a fallback.
- Rate and cost monitoring catch scraping, model theft, runaway agents, DoS and leaked keys.
- Audits cover hallucinations, accuracy, bias and fairness, and access.
Prompt logs are themselves sensitive data. Logging every prompt in clear text without sanitization creates a new store of secrets and personal data for an attacker to steal.
Common mistakes
Monitoring availability only. A model can be up and still be leaking data or giving wrong answers. Monitor content, not just uptime.
Treating cost as a finance problem. A sudden cost rise is a security signal and should alert the security team as well.
Trusting a confident answer. A high confidence score does not prove an answer is true, so hallucination audits still sample confident responses.
Check yourself
What two sides of an exchange does prompt monitoring cover?
The query sent to the model and the response the model returns.
What is log sanitization?
Removing or masking secrets, credentials and personal data from logs before they are stored.
Why can an unexpected rise in AI cost indicate an attack?
It may come from a leaked key, scraping, model theft by mass querying or an agent stuck in a loop.
Topic 5 of 7 in Securing AI Systems
Test yourself on Securing AI Systems
The full CompTIA SecAI+ CY0-001 question bank has 654 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by CompTIA. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.