Security Operations · Topic 9 of 10

AI in Security Operations

How AI and machine learning help detect and triage threats, plus the risks of prompt injection, data leakage and poisoning, and the rules for safe use.

6 min

In a nutshell

AI is a helper that suggests, and the analyst remains accountable for decisions.
Common uses are anomaly detection, triage, classification, search, summaries and drafting.
Supervised learning uses labelled data, and unsupervised learning looks for patterns or outliers.

What it is

Artificial intelligence (AI) in security covers software that learns patterns from data and uses them to make predictions or produce content. Machine learning (ML) is the part of AI that learns from examples, and a large language model (LLM) is an ML model trained on large amounts of text that can answer questions and write text or code.

AI helps defenders work faster, but it is also a new system to protect and a tool for attackers. The exam looks at all three sides: use cases, risks and governance.

How it works

Defensive use cases

Use case	What AI does
Anomaly detection	Learns normal behaviour and flags outliers, as in user and entity behaviour analytics
Alert triage	Groups, ranks and summarises alerts so humans see the important ones first
Malware and phishing classification	Scores files or messages by learned features rather than exact signatures
Natural language search	Turns a plain question into a query for the log platform
Report and ticket drafting	Summarises an incident timeline or writes first drafts of documents
Script and rule assistance	Suggests detection rules, parsers or scripts for the analyst to check
Intelligence summarising	Condenses long reports and extracts indicators

Two broad ML styles appear in security. Supervised learning trains on labelled examples (known good and bad), and unsupervised learning finds structure or outliers in data without labels.

Comparing artifacts with AI

An assistant can compare two scripts, two configuration files, a known good and a suspicious file, or two sets of logs, and explain the differences in plain language. It may point out a new outbound address, disabled logging or an added administrator account.

Check the answer against a deterministic tool, because a model can miss a change or invent one. A hash comparison shows whether two files are identical, and a diff lists exact changes.

12c12
< PermitRootLogin no
---
> PermitRootLogin yes

This diff output shows that line 12 of a Secure Shell (SSH) server configuration changed from no to yes. The AI summary should say the same, and the analyst should sanitise secrets before sharing the files with an external service.

Attacker use of AI

Attackers use AI to write convincing phishing messages, copy a voice or face in a deepfake, and speed up reconnaissance and malware changes. The defender's response is mostly the same controls with more emphasis on verification, such as call back procedures for payment requests and strong identity checks.

Risks of using AI

Hallucination: a model can state false details with confidence, so an invented indicator or fix could mislead an investigation.
Data leakage: pasting logs, credentials or customer data into an external AI service can expose it.
Prompt injection: hidden or crafted text in the input makes the model ignore its instructions or reveal data. An LLM reading an email or a web page can be steered by instructions inside that content.
Data poisoning: an attacker corrupts training data, so the model learns wrong behaviour.
Evasion: inputs are shaped to slip past a classifier.
Model theft or inversion: an attacker copies the model or extracts training data from it.
Bias and drift: poor or old training data produces unfair or stale results, and accuracy drops as behaviour changes.
Over reliance: analysts stop checking the output, and skills fade.
Lack of explainability: if nobody can say why the model flagged something, the finding is hard to defend.

Governance

Governance is the set of rules and oversight that keep AI use safe, lawful and accountable.

Policy: an acceptable use rule that says which tools are approved and what data may be entered.
Data classification: sensitive data stays out of unapproved services.
Human oversight: a person reviews output before high impact action.
Validation: test accuracy and false positive rates before and after deployment.
Inventory and vendor review: know which AI systems and third parties are in use, and assess their security and privacy terms.
Logging and audit: record prompts, outputs and decisions so they can be reviewed.
Access control: restrict who can use or change models, prompts and training data.
Compliance: follow privacy and sector rules.

Published frameworks help structure this work. Examples are the NIST (National Institute of Standards and Technology) AI Risk Management Framework and the OWASP (Open Worldwide Application Security Project) guidance for LLM application risks.

A safe pattern for analysts

Allowed:   ask an approved assistant to explain an unfamiliar log field, using sanitised sample text
Allowed:   have it draft a summary that you then check against the raw evidence
Not safe:  paste raw customer records or secrets into a public chatbot
Not safe:  run a generated script on production without review and testing

What to remember for the exam

AI is a helper that suggests, and the analyst remains accountable for decisions.
Common uses are anomaly detection, triage, classification, search, summaries and drafting.
Supervised learning uses labelled data, and unsupervised learning looks for patterns or outliers.
Hallucination means plausible but false output.
Prompt injection manipulates a model through crafted input, and data poisoning corrupts its training data.
Sending sensitive data to an unapproved AI service is a data leakage risk.
Governance includes policy, classification, human review, validation, vendor review, logging and access control.
Attackers use AI for phishing, deepfakes and faster reconnaissance.
Validate AI generated scripts and rules before using them in production.

Watch out

Treat AI output as untrusted until verified. A confident answer can still be wrong, and a pasted secret cannot be taken back.

Common mistakes

Assuming AI results are facts. They are predictions or drafts, so check them against evidence.

Mixing up prompt injection and data poisoning. Injection happens at use time through input, while poisoning happens earlier in the training data.

Ignoring the data that leaves the organisation. Inputs to a hosted service may be stored or used by the provider, depending on the terms.

Believing AI replaces analysts. It speeds up routine work, but judgement, context and accountability stay with people.

Skipping ongoing testing. Models drift as networks and attacks change, so accuracy must be rechecked.

Check yourself

What is a hallucination?
A model output that sounds confident but is false or invented.
What is the difference between prompt injection and data poisoning?
Prompt injection manipulates a model through crafted input when it is used, while data poisoning corrupts the training data beforehand.
Name one governance control for analysts using an AI assistant.
An acceptable use policy that limits what data may be entered, with human review of the output.

Topic 9 of 10 in Security Operations

Test yourself on Security Operations

The full CompTIA CySA+ CS0-004 question bank has 654 questions, with timed practice exams and an explanation for every answer.

Subscribe Try Sample Exam

This study guide is independent and is not affiliated with, endorsed by, or sponsored by CompTIA. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.