AI Auditing Tools and Techniques · Topic 2 of 5
AI Audit Testing and Sampling
Testing AI controls: design versus operating effectiveness, walkthroughs, reperformance, attribute and variable sampling, and full population tests.
- 5 min
- Includes a diagram
- Test design before operating effectiveness.
- Inquiry alone is never sufficient evidence.
- Attribute sampling measures rates; variable sampling measures amounts.
What it is
Audit testing is how the auditor gathers proof that controls are well designed and work, and that outcomes are correct. Sampling lets the auditor reach a conclusion about a large population, such as thousands of model decisions, by examining a subset.
The outline asks the auditor to evaluate the design and effectiveness of controls specific to AI. The core audit ideas do not change; what changes is what is tested. With AI, the population may be decisions, model versions, prompts, retraining events or monitoring alerts.
How it works
Design versus operating effectiveness
| Test | Question | AI example |
|---|---|---|
| Design effectiveness | If it works as described, would the control address the risk? | Does the monitoring procedure define fairness thresholds and escalation? |
| Operating effectiveness | Did it work consistently through the period? | Were fairness breaches in the year escalated and acted on? |
Design comes first. If a control is badly designed, testing its operation wastes effort; the auditor reports the design gap.
Testing techniques
| Technique | Use in AI audits |
|---|---|
| Inquiry | Understand the process; never sufficient alone |
| Observation | Watch a reviewer handle AI recommendations |
| Inspection | Review approvals, validation reports, change records |
| Walkthrough | Trace one decision or one release end to end |
| Reperformance | Rerun a control, such as recalculating a fairness metric from the data |
| Recomputation and analytics | Analyse the full set of decisions for patterns |
Compliance tests check that controls operate. Substantive tests check that outcomes or data are correct, such as whether a sample of AI decisions was right. Strong controls allow less substantive testing.
Testing the model versus the controls
The auditor is usually not the model validator. Two approaches combine well:
| Approach | What it gives |
|---|---|
| Test controls around the model | Assurance that validation, change, monitoring and oversight work |
| Test outcomes directly | Independent evidence on a sample of decisions or a reperformed metric |
Where outcome testing needs specialist skill, the auditor can use an expert, or review the independent validation function's work and decide how far to rely on it.
Sampling methods
| Method | Use |
|---|---|
| Statistical sampling | Random or systematic selection; results can be projected with measured confidence |
| Judgmental sampling | Auditor selects items, such as high-risk cases; cannot measure sampling risk |
| Attribute sampling | Measures a rate, such as how often approvals were missing |
| Variable sampling | Estimates an amount, such as the value of incorrect payouts |
| Discovery sampling | Finds at least one example of a rare event |
| Stop-or-go sampling | Stops early when few errors are expected and none are found |
Sample size rises with higher confidence and higher expected error, and falls with higher tolerable error.
Sampling AI populations
| Issue | Approach |
|---|---|
| Decisions change by model version | Define the population by version and period; sample each version that was live |
| Risk differs by group or segment | Stratify by group, region or decision type so small groups are tested |
| Rare harmful outcomes | Use discovery sampling or full population analytics |
| Generative output varies | Use fixed evaluation prompts and repeated runs, and record settings |
Before sampling, confirm the population is complete, for example by reconciling decision log counts to the source system.
Full population testing
Because AI decisions are logged, the auditor can often test every decision with analytics: for example, compare approval rates by group across the full year. This removes sampling risk for that test, though data quality must still be confirmed.
Evaluating results
Each exception is examined for cause. One missing approval may be an isolated error; a pattern across a model's releases is a control failure. Results are compared with tolerable error, and conclusions documented in work papers that another auditor could follow.
A worked example
An auditor tests whether retrained models were approved before deployment. The registry shows 48 deployments in the year. Using attribute sampling, the auditor selects 25 and finds three deployed before approval, all by the same team using an emergency route.
The auditor concludes the control is not operating effectively and examines all emergency deployments, a targeted extension of the test to understand the cause.
How the exam asks about this
- "Which test BEST confirms a fairness monitoring control operated during the year?" Inspect evidence that breaches during the period were escalated and resolved, or reperform the metric.
- "An auditor wants to make sure small customer groups are tested. Which technique is BEST?" Stratified sampling by group.
- "Which sampling method is BEST to detect whether any AI decision used a prohibited input?" Discovery sampling or full population analytics.
What to remember for the exam
- Test design before operating effectiveness.
- Inquiry alone is never sufficient evidence.
- Attribute sampling measures rates; variable sampling measures amounts.
- Stratify AI populations by version, group and segment.
- Confirm population completeness before sampling or analytics.
A sample drawn from one model version does not support a conclusion about the whole year if other versions were live. Define the population by version and period first.
Common mistakes
Testing operation of a poorly designed control. Report the design gap first.
Projecting judgmental samples. Only statistical samples support objective projection to the population.
Ignoring analytics. Full population tests are often possible with AI decision logs and give stronger evidence.
Check yourself
What is the difference between design and operating effectiveness?
Design asks whether the control would address the risk if it works as described; operating effectiveness asks whether it actually worked throughout the period.
Why stratify a sample of AI decisions?
To make sure smaller or higher-risk groups, segments or model versions are adequately tested rather than swamped by the majority.
When is discovery sampling appropriate?
When the auditor needs to find at least one example of a rare but serious event, such as use of a prohibited input.
Topic 2 of 5 in AI Auditing Tools and Techniques
Test yourself on AI Auditing Tools and Techniques
The full ISACA AAIA question bank has 750 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by ISACA. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.