Basic AI Concepts Related to Cybersecurity · Topic 4 of 4
Security Across the AI Life Cycle
Secure each AI life cycle stage, from business use case and data collection to deployment and monitoring, with human-in-the-loop oversight and validation.
- 6 min
- Includes a diagram
- Security starts at the business use case, not at deployment.
- Data collection must check the trustworthiness of sources and the authenticity of records.
- Evaluation tests accuracy, bias and resistance to attack before release; validation confirms behaviour in the real environment.
What it is
The artificial intelligence (AI) life cycle is the sequence of stages an AI system passes through, from the first business idea to monitoring and improvement in production. Security has to be built into every stage, because a weakness introduced early, such as untrusted training data, cannot be fixed by a firewall at the end.
Human-centric AI design keeps people involved where judgement and accountability matter. It is the counterweight to automation: AI proposes, and humans review, approve or correct.
The last stage loops back to the first. Feedback from production drives new data, retraining and sometimes a new use case.
How it works
Stage by stage
| Stage | Key question | Security focus |
|---|---|---|
| Business use case | What problem does AI solve, and is AI the right tool? | Alignment with corporate objectives, risk appetite, legal limits, data needed |
| Data collection | Where will the data come from? | Trustworthiness of sources and authenticity of records; consent and licensing |
| Data preparation | Is the data clean, labelled and safe to use? | Cleansing, balancing, removing or masking sensitive data, recording lineage |
| Model development or selection | Build our own, fine-tune, or use a third-party model? | Vetting model sources, scanning model files, secure training environment |
| Model evaluation | Does it meet accuracy, fairness and safety targets? | Testing against held-out data, bias tests, adversarial and red team tests |
| Deployment | How is it released and reached? | Access controls, gateway, guardrails, secrets management, change control |
| Validation | Does it behave as expected in the real environment? | Confirm results on live data before full rollout, check guardrails work |
| Monitoring and maintenance | Is it still accurate and safe? | Watch for drift, abuse, cost spikes and attacks; patch dependencies |
| Feedback and iteration | What should change? | Verify feedback before it is used to retrain, so attackers cannot skew the model |
The business use case comes first
Every AI project should start with a documented reason that aligns with corporate objectives. A use case defines what data is needed, who uses the system and what harm a wrong answer could cause.
That harm estimate sets how much control the project needs. A tool that drafts internal meeting notes needs less oversight than one that approves refunds or blocks user accounts.
Trustworthy and authentic data
Trustworthiness asks whether a source is reliable and appropriate, such as an official feed versus an anonymous forum. Authenticity asks whether a specific record is genuine and unaltered.
Collecting from scraped public sites, purchased data sets or user uploads raises both questions. Signed sources, provenance records and verification checks address them.
Develop or select
Many organisations select a pre-trained model rather than building one. Selection is a supply chain decision: check the publisher, licence, documentation and known weaknesses, and scan model files before loading them.
Monitoring and drift
Model drift is the drop in performance as real-world data changes away from the training data. Attack techniques change, so a detection model that was accurate at launch can quietly decline.
Monitoring compares live accuracy, confidence and input patterns with the baseline. A significant change triggers review and, if needed, retraining.
Human-centric design principles
| Principle | Meaning | Example |
|---|---|---|
| Human-in-the-loop | A person must act on or approve each decision before it takes effect | An analyst approves every account lockout an AI recommends |
| Human oversight | People supervise the system as a whole and can intervene or shut it down | A review board watches metrics and can disable an agent |
| Human validation | People check outputs for correctness, especially samples or high-impact cases | Reviewers grade a weekly sample of AI-written incident summaries |
The level of human involvement should match the impact of a wrong decision. Low-risk, reversible actions can run with oversight only, while high-impact actions need a human in the loop.
A worked example
A bank plans an AI model that suggests whether to block suspicious card payments. The use case links to the objective of reducing fraud losses, and the team collects transaction history only from its own verified systems.
During preparation, card numbers are tokenised and fraud cases are balanced. After evaluation, the model goes live in a shadow mode for validation, then blocks only low-value payments automatically, while high-value blocks wait for an analyst's approval.
How the exam asks about this
- "Before collecting training data, a project team confirms the AI tool supports a stated business goal. Which of the following life cycle activities is this?" Defining the business use case and its alignment with corporate objectives.
- "An AI system recommends disabling user accounts, and policy requires an analyst to approve each action. Which of the following principles does this describe?" Human-in-the-loop.
- "A malware classifier that performed well last year now misses many new samples. Which of the following life cycle stages should have detected this?" Monitoring and maintenance, which watches for model drift.
What to remember for the exam
- Security starts at the business use case, not at deployment.
- Data collection must check the trustworthiness of sources and the authenticity of records.
- Evaluation tests accuracy, bias and resistance to attack before release; validation confirms behaviour in the real environment.
- Monitoring watches for drift, abuse and cost; feedback must be verified before it is used to retrain.
- Human-in-the-loop means approval per decision; human oversight means supervising the system as a whole.
Feedback loops are an attack surface. If user ratings or corrections flow straight back into training without checks, an attacker can gradually skew the model.
Common mistakes
Treating evaluation and validation as the same step. Evaluation measures the model against test criteria before release, while validation confirms it works correctly in its deployed setting.
Putting a human in the loop for everything. Approval for every low-risk action causes fatigue and rubber-stamping. Match the level of human involvement to the impact.
Forgetting retirement and maintenance. Models, dependencies and data stores need patching, review and eventual decommissioning, including deleting data that is no longer needed.
Check yourself
What is model drift?
A decline in model performance because real-world data has changed away from the data the model was trained on.
What is the difference between trustworthiness and authenticity in data collection?
Trustworthiness is whether the source is reliable and appropriate, while authenticity is whether a specific record is genuine and unaltered.
When should a human be in the loop rather than only providing oversight?
When a wrong decision would have high impact or be hard to reverse, so each action needs human approval first.
Topic 4 of 4 in Basic AI Concepts Related to Cybersecurity
Test yourself on Basic AI Concepts Related to Cybersecurity
The full CompTIA SecAI+ CY0-001 question bank has 654 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by CompTIA. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.