AI Governance and Risk · Topic 1 of 6
AI Models, Considerations and Requirements
How an IT auditor evaluates AI models and requirements: model types, generative AI, agents, business case, architecture fit and impact on people.
- 5 min
- Know the model type; it tells you how the system can fail.
- AI output is probabilistic, so test against agreed thresholds over time.
- Requirements must include performance, explainability, oversight and data, not just function.
What it is
Before the auditor can judge controls over an AI system, the auditor must understand what kind of model it is, what it is meant to do and what requirements it was built against. Different model types fail in different ways, so they need different controls and different audit tests.
This sub-domain is also where the auditor advises stakeholders on whether an AI solution fits the strategy, the enterprise architecture and the people it affects. The auditor does not choose the model. The auditor evaluates whether the organization chose it for documented, defensible reasons.
How it works
Model types the auditor should recognise
| Type | How it learns | Typical use | Audit concern |
|---|---|---|---|
| Supervised learning | From labelled examples | Credit scoring, fraud detection, classification | Quality and bias of the labels |
| Unsupervised learning | Finds patterns without labels | Clustering customers, anomaly detection | Hard to validate; results need business interpretation |
| Reinforcement learning | From rewards for actions | Pricing, scheduling, robotics | The reward may drive unintended behaviour |
| Deep learning | Many layered neural networks | Image, speech, language | Low explainability |
| Generative AI and large language models | Pretrained on large data, often adapted | Drafting, summarising, chat, code | Fabricated output, data leakage, prompt injection |
| Rules engines | Not learning; coded logic | Eligibility checks | Often wrongly labelled as AI |
Many enterprise uses are not a model alone. Retrieval augmented generation (RAG) connects a language model to the organization's documents, and AI agents let a model call tools and take actions. Each added connection widens the set of data, permissions and outputs to audit.
Deterministic versus probabilistic
Traditional applications give the same output for the same input, so a single test proves a rule. AI models are probabilistic: they give a likely answer, can be wrong with confidence, and may change behaviour when data changes. The auditor therefore asks for performance thresholds agreed in advance and evidence that they are met over time, not a single successful demonstration.
Requirements the auditor evaluates
| Requirement | What good looks like |
|---|---|
| Business | A stated objective, owner, expected benefit and measure of success |
| Functional | What the system must decide or produce, and what it must never do |
| Performance | Accuracy or error thresholds, set before build, linked to business harm |
| Explainability | The level of explanation users, customers or regulators need |
| Human oversight | Where a person reviews, overrides or stops the system |
| Data | Sources, rights to use, quality and privacy limits |
| Non-functional | Security, availability, latency, cost, logging |
| Regulatory | Classification under applicable law and the duties that follow |
Alignment with enterprise architecture
An AI solution should fit the enterprise architecture: approved platforms, data stores, identity services, integration patterns and security standards. The auditor checks that the architecture review board or an equivalent reviewed the solution, and that it does not create a shadow data copy, a new unmanaged identity or an unsupported integration.
Build, buy or adapt is part of this decision. A vendor model brings speed but less visibility; an in-house model brings control but needs skills. Either way the requirement is the same: the organization must be able to evidence how the model meets its requirements.
Impact on systems, environment and people
| Impact area | Questions the auditor asks |
|---|---|
| System interactions | What upstream data feeds it, and which downstream systems act on its output without review? |
| Humans | Who is affected by its decisions, and can they question or appeal them? |
| Workforce | Which roles change, and is there a plan for retraining? |
| Environment | Has the energy and compute cost been weighed against the benefit? |
| Organization | Does reliance on one model or vendor create concentration risk? |
AI decision-making systems need the most attention. A model that recommends to a trained person is lower risk than one that decides automatically about customers, employees or citizens.
A worked example
A retailer plans a model that automatically approves or declines customer refunds. The business case shows cost savings but no error threshold, no definition of who reviews declined refunds and no link to the customer complaints process.
The auditor reports that the requirements are incomplete. The recommendation is that management define performance thresholds tied to customer harm, a human review path for declines and an owner for the model before approval. The auditor does not set the threshold.
How the exam asks about this
- "What should an IS auditor review FIRST when asked to advise on a proposed AI solution?" The business case and requirements, to confirm alignment with organizational objectives.
- "Which is the GREATEST concern for an AI system that makes decisions about customers automatically?" The absence of human oversight and a way for affected people to challenge decisions.
- "An AI solution was deployed outside the approved architecture. What is the auditor's MAIN concern?" Unmanaged integration, data and security risk, because the solution bypassed architecture review.
What to remember for the exam
- Know the model type; it tells you how the system can fail.
- AI output is probabilistic, so test against agreed thresholds over time.
- Requirements must include performance, explainability, oversight and data, not just function.
- AI solutions must pass enterprise architecture review like any other system.
- Automated decisions about people carry the highest impact.
A successful demonstration is not evidence that a model meets its requirements. Ask for measured results on representative data against thresholds set before the model was built.
Common mistakes
Calling every automated rule AI. Rules engines are deterministic and are audited like ordinary application controls.
Accepting a business case with benefits but no measures. Without a measure of success, nobody can later show the model delivered value or decide to retire it.
Focusing only on the model. Data feeds, integrations, tools an agent can call and downstream actions are often where the real risk sits.
Check yourself
Why must AI performance thresholds be set before the model is built?
So results are judged against a standard linked to business harm, not adjusted after the fact to match whatever the model achieved.
What is the main extra risk when a language model is turned into an agent?
It can take actions through tools and permissions, so errors or manipulated inputs can cause real changes, not just wrong text.
Why does the auditor check enterprise architecture alignment for AI?
To make sure the solution uses approved platforms, identity and data patterns, and does not create unmanaged data copies or integrations.
Topic 1 of 6 in AI Governance and Risk
Test yourself on AI Governance and Risk
The full ISACA AAIA question bank has 750 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by ISACA. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.