AI Auditing Tools and Techniques · Topic 4 of 5
Audit Data Quality and Data Analytics
Data quality and analytics in AI audits: quality dimensions, completeness checks, CAATs, continuous auditing, and using AI tools in audit with care.
- 4 min
- Validate completeness and accuracy before analytics.
- Analytics on full populations remove sampling risk but not data quality risk.
- Continuous auditing is the auditor's; continuous monitoring is management's.
What it is
Audit data analytics uses software to examine large datasets, often whole populations, to find exceptions, trends and risks. Its value depends on data quality: analytics on incomplete or wrong data produce confident but false conclusions.
This sub-domain covers both sides of AI in audit. The auditor analyses data about AI systems, and the audit function increasingly uses AI to analyse data, which brings opportunities and risks the auditor must evaluate.
How it works
Data quality dimensions
| Dimension | Audit check |
|---|---|
| Completeness | Record counts and totals reconcile to the source |
| Accuracy | Sample values agree to source documents or systems |
| Validity | Values follow expected formats and ranges |
| Consistency | The same item has the same value across systems |
| Timeliness | Data covers the right period and is current |
| Uniqueness | No unexpected duplicates |
The auditor validates data quality before relying on analytics. Completeness is the first check, because missing records cannot produce exceptions.
Analytics techniques
| Technique | AI audit use |
|---|---|
| Filtering and exception reports | Decisions made by unapproved model versions |
| Matching and joins | Deployments with no matching change approval |
| Stratification and ageing | Open model findings by age and severity |
| Trend analysis | Drift in approval rates or alert volumes over time |
| Group comparison | Outcome rates by group to look for disparity |
| Duplicate and gap tests | Missing log periods or repeated records |
| Benford's law and outlier tests | Unusual patterns in amounts |
Computer-assisted audit techniques (CAATs) include generalised audit software, scripts and data tools. The auditor controls the scripts used, keeps them with the work papers and checks they do what was intended.
Continuous auditing and monitoring
Continuous monitoring is management's ongoing automated checking. Continuous auditing is the auditor's own frequent, automated testing. For AI, continuous auditing might rerun checks on model versions, approvals and fairness metrics every month. The auditor can rely on management's monitoring only after testing it.
Opportunities of AI in the audit process
| Area | Example |
|---|---|
| Planning | Summarise policies, prior reports and risk data |
| Execution | Classify transactions, read contracts, flag anomalies across full populations |
| Documentation | Draft work paper summaries |
| Reporting | Draft findings from structured results |
These can widen coverage, from samples to full populations, and speed up routine work.
Risks of AI in the audit process
| Risk | Control the audit function needs |
|---|---|
| Inaccurate or fabricated output | Validate results against source data; human review of every conclusion |
| Confidentiality | Approved tools only, with contract terms on data use; no client data in public tools |
| Lack of explainability | Use methods whose results can be explained and reproduced |
| Bias in the tool | Test the tool on known data before relying on it |
| Over-reliance | Professional judgement remains with the auditor |
| Documentation | Record tool, version, inputs and how output was checked |
The audit function should have its own policy for AI use, approved tools, training and quality review. When the outline asks the auditor to evaluate impacts, opportunities and risk when integrating AI within the audit process, this is what it means: treat audit's AI like any other AI system.
A worked example
An audit team uses an AI tool to classify 50,000 expense claims and flag those that breach policy. The tool flags 400. Before reporting, the auditor tests the tool on 200 claims already classified by hand and finds it misses claims written in a second language.
The auditor reports results with this limitation stated, reviews the second-language claims separately, and records the tool, version, validation test and results in the work papers.
How the exam asks about this
- "What should the auditor do FIRST before running analytics on AI decision logs?" Confirm the data is complete and accurate, for example by reconciling counts to the source.
- "Which is the GREATEST risk of using a public generative AI tool to summarise audit evidence?" Disclosure of confidential or personal data.
- "An AI audit tool identifies exceptions across a full population. What must the auditor do before relying on them?" Validate the tool's output against source data and document the check.
What to remember for the exam
- Validate completeness and accuracy before analytics.
- Analytics on full populations remove sampling risk but not data quality risk.
- Continuous auditing is the auditor's; continuous monitoring is management's.
- AI in audit needs policy, approved tools, validation and documentation.
- The auditor, not the tool, owns every conclusion.
AI tools can produce fluent, convincing output that is wrong. An AI-generated finding that has not been traced to source evidence is not audit evidence.
Common mistakes
Skipping data validation because the data came from a system. Extraction errors and filters can still drop records.
Using public AI tools with client data. Confidentiality and contract terms come first.
Not documenting tool use. Reviewers and regulators must be able to see how AI contributed and how it was checked.
Check yourself
Why is completeness the first data quality check before analytics?
Missing records cannot produce exceptions, so incomplete data hides problems and gives false comfort.
What is the difference between continuous auditing and continuous monitoring?
Continuous auditing is the auditor's own automated testing; continuous monitoring is management's ongoing automated checking.
What should be documented when AI is used in an audit?
The tool and version, the inputs, how the output was validated and the auditor's conclusion.
Topic 4 of 5 in AI Auditing Tools and Techniques
Test yourself on AI Auditing Tools and Techniques
The full ISACA AAIA question bank has 750 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by ISACA. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.