AI Operations · Topic 5 of 8
Testing Techniques for AI Solutions
AI testing for auditors: holdout and cross-validation, confusion matrix metrics, bias, robustness, explainability, red teaming and generative AI tests.
- 6 min
- Precision is about false alarms; recall is about misses.
- Accuracy misleads on imbalanced data.
- Fairness measures must be chosen and justified per use.
What it is
AI testing checks whether a model performs well enough, fairly enough and safely enough for its purpose. Ordinary software testing asks "does it do what the specification says?" AI testing also asks "how often is it wrong, for whom, and how does it behave under stress or attack?"
The auditor may not run these tests personally. The auditor evaluates whether the right tests were performed for the risk, on appropriate data, by people independent enough, against thresholds set in advance, and whether failures stopped release.
How it works
Performance testing
| Technique | Purpose |
|---|---|
| Holdout test set | Evaluate on data the model never saw in training |
| Cross-validation | Rotate training and test splits to check stability |
| Backtesting | Run the model on past periods to see how it would have performed |
| Benchmark comparison | Compare with the current model, a simple baseline or human performance |
Classification metrics
| Metric | Meaning | Matters when |
|---|---|---|
| Accuracy | Share of all predictions that are correct | Classes are balanced |
| Precision | Of items flagged positive, how many really are | False alarms are costly |
| Recall | Of real positives, how many were caught | Misses are costly, such as fraud or disease |
| F1 score | Balance of precision and recall | Both matter |
| False positive and false negative rates | Error types from the confusion matrix | Different harms per error type |
Accuracy misleads on imbalanced data. If 1% of transactions are fraud, a model that flags nothing is 99% accurate and useless. The auditor checks that the chosen metric matches the business harm.
Overfitting, underfitting and ROC curves
| Term | Meaning | What the auditor looks for |
|---|---|---|
| Overfitting | The model learns the training data too closely, including its noise, so it scores well in training but poorly on new data | A large gap between training and holdout results |
| Underfitting | The model is too simple to capture the pattern, so it scores poorly on both training and test data | Low results everywhere, often a sign of weak features or too little training |
| ROC curve and AUC | The ROC curve plots the true positive rate against the false positive rate across decision thresholds; the area under it (AUC) summarizes how well the model separates the classes, with 0.5 no better than chance and 1.0 perfect | Whether the chosen threshold matches the business cost of false positives and false negatives |
Fine-tuning and transfer learning adapt a model already trained on one task to a new one with less data. The auditor treats the adapted model as a new model: it needs its own testing, documentation and approval.
Bias and fairness testing
Fairness testing compares outcomes or error rates across relevant groups. Different fairness measures can conflict, so the organization must choose and justify which applies to each use. The auditor checks that groups were defined, the measure was justified, results were within threshold, and failures led to action.
Robustness and stress testing
Robustness tests check behaviour with noisy, missing, unusual or extreme inputs, and with data from new conditions. Adversarial testing checks whether small deliberate changes to inputs fool the model. These tests show how the model fails, which matters as much as how often it succeeds.
Explainability testing
Explanation tools such as SHAP and LIME estimate which features drove a prediction. The auditor checks that the main drivers make business sense and that no prohibited or proxy feature dominates. An explanation that contradicts domain knowledge is a reason to investigate.
Generative AI evaluation
| Test | What it checks |
|---|---|
| Groundedness and factuality | Answers are supported by approved sources, not fabricated |
| Evaluation sets | Fixed sets of prompts with expected answers, rerun on every change |
| Safety and content testing | Refusal of prohibited requests, no harmful output |
| Red teaming | Skilled testers try to make the system misbehave or leak data |
| Regression testing | A prompt or model change did not break earlier behaviour |
Red teaming is a structured, authorised exercise with defined scope and rules. The auditor checks that findings were tracked to closure, not just recorded.
Integration and user acceptance testing
The model is one component. Integration testing checks data flows, downstream actions and fallbacks when the model is unavailable. User acceptance testing confirms business users can work with the outputs and that human oversight steps function.
Independence and data
Test data must be representative of production, and testers for high-risk models should be independent of the developers. Test results should be reproducible: the auditor should be able to see the data version, model version and code used.
A worked example
A lender's validation report shows 94% accuracy for a credit model. The auditor notes that defaults are 6% of the data, so a model that predicts no default for everyone would also score 94%. The report shows no recall figure and no breakdown by applicant group.
The auditor concludes that testing does not show the model is fit for purpose. The recommendation is to report recall, false negative rate and fairness metrics against predefined thresholds before the model is relied on.
How the exam asks about this
- "A fraud model is rarely missing fraud but flags many legitimate transactions. Which metric is LOW?" Precision.
- "What is the auditor's GREATEST concern when test results report only overall accuracy on imbalanced data?" Accuracy can hide poor detection of the minority class that matters.
- "Which is the BEST way to test whether a generative AI assistant can be manipulated into revealing restricted data?" An authorised red team exercise with defined scope.
What to remember for the exam
- Precision is about false alarms; recall is about misses.
- Accuracy misleads on imbalanced data.
- Fairness measures must be chosen and justified per use.
- Red teaming is authorised, scoped and tracked to closure.
- Thresholds are set before testing; failures must block release.
Test results only count if the release decision depended on them. Ask for evidence that a failed test stopped or changed a release, not just that tests ran.
Common mistakes
Mixing up precision and recall. Precision asks "of what I flagged, how much was right?"; recall asks "of what was there, how much did I catch?"
Testing on unrepresentative data. Good results on old or narrow data say little about production.
Treating red team reports as the end. Without remediation tracking, known weaknesses stay open.
Check yourself
What does recall measure?
The share of actual positive cases the model correctly identified, so low recall means many misses.
Why is accuracy a poor metric for a fraud model?
Fraud is rare, so a model can score very high accuracy while catching little or no fraud.
What do SHAP and LIME help an auditor evaluate?
Which features drive a model's predictions, so the auditor can check they make business sense and no prohibited or proxy feature dominates.
Topic 5 of 8 in AI Operations
Test yourself on AI Operations
The full ISACA AAIA question bank has 750 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by ISACA. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.