AI Audit Planning and Design

What it is

Audit planning decides which AI systems to audit, what each audit must achieve and how it will be done. Audit design turns that into an audit program: the objectives, scope, criteria, tests and evidence needed.

Planning an AI audit follows the same professional standards as any IS audit, such as ISACA's ITAF professional practices framework for IS audit and assurance. What changes is the subject: systems that learn, change with data, rely on vendors and can harm people outside the organization.

How it works

Building the AI audit universe

The AI inventory is the starting point for the audit universe. Because inventories are often incomplete, the auditor supplements it with procurement data, architecture records, vendor lists and discussions with business units. AI embedded in purchased software is easy to miss.

Risk-based selection

The auditor rates AI systems on these factors and directs effort to the highest risk. Management's own risk ratings are an input, not a substitute for the auditor's assessment.

Engagement objectives and scope

Objectives state what the audit will conclude on, for example "whether controls over the credit model provide reasonable assurance that decisions are accurate, fair and compliant". Scope sets which systems, lifecycle stages, locations, vendors and period are included.

Criteria

Criteria are the standard against which findings are measured: laws, regulations, adopted frameworks such as NIST AI RMF or ISO/IEC 42001, contracts and internal policy. They should be agreed with management during planning.

Understanding the system

Before designing tests, the auditor needs to understand how the AI system works: its purpose, data flows, model type, integrations, human oversight points and key controls. Sources include model cards, datasheets, architecture diagrams, walkthroughs and interviews with owners and developers. A walkthrough of one decision from input to outcome is often the fastest way in.

Competence and independence

AI audits may need skills the team lacks, such as data science or model validation. Options include training, hiring or using an expert. When using an expert, the auditor remains responsible for the conclusion and must assess the expert's competence and independence.

Independence also matters if internal audit advised on the AI program's design. An auditor who designed a control should not audit it.

Audit program

The audit program lists, for each risk and control, the test steps, the sample approach, the evidence required and who performs it. For AI, it should include tests of both controls (approvals, monitoring, change) and, where in scope, the model's outcomes (reperformance or analysis of decisions).

Using AI in audit planning

AI tools can help planning: summarising policies and prior reports, scanning large volumes of documents for AI use, or analysing data to highlight risk areas. The auditor evaluates the impacts, opportunities and risks of doing so: confidentiality of client data entered into the tool, accuracy of its output, and whether its use is documented and approved under the audit function's own policy.

A worked example

The chief audit executive asks for an AI audit plan. The AI inventory lists 12 systems. Procurement data shows a further four vendor tools with AI features, including an HR screening tool. The auditor rates all 16, and the HR screening tool scores highest: automated ranking of candidates, personal data, a listed high-risk use and no prior review.

The plan places it first, with criteria drawn from applicable law, the organization's AI policy and its adopted framework. A data scientist from outside the HR function joins as an expert.

How the exam asks about this

• "What should an auditor do FIRST when planning an AI audit program for the year?" Identify the AI audit universe, using the inventory and other sources to confirm completeness.
• "Which factor MOST increases the audit priority of an AI system?" Automated decisions with significant impact on people.
• "An audit uses an outside data scientist. Who is responsible for the audit conclusion?" The auditor, who must assess the expert's competence and independence.