AI Operations · Topic 3 of 8
Change and Configuration Management for AI
Auditing AI change and configuration management: retraining as change, prompt and parameter changes, vendor model updates, baselines, rollback and drift.
- 5 min
- Includes a diagram
- Retraining, threshold, prompt, retrieval and vendor changes are all changes.
- Test changes against the approved baseline, including fairness.
- Segregate training, approval and deployment duties.
What it is
Change management makes sure changes reach production only after they are requested, assessed, tested and approved. Configuration management records the approved state of each system so changes can be detected and reversed. Both are core IT general controls the auditor already knows.
AI widens what counts as a change. A model can change behaviour without any code change: new training data, a new prompt, a changed threshold or a vendor's silent model update. The auditor evaluates whether the change and configuration programs recognise all of these.
How it works
What counts as an AI change
| Change | Why it matters |
|---|---|
| Retraining on new data | Can shift accuracy, bias and behaviour |
| New model version or architecture | New behaviour, new failure modes |
| Decision threshold change | Alters approval or alert rates directly |
| Feature changes | Adding or removing inputs changes what the model relies on |
| System prompt or prompt template change | Changes how a generative model behaves and what it refuses |
| Retrieval source change | Changes what knowledge a generative AI draws on |
| Tool or permission change for an agent | Changes what actions it can take |
| Vendor model update | Behaviour changes without any action by the organization |
| Guardrail or filter change | Affects safety controls |
The auditor checks that policy defines these as changes and that each type flows through the change process in proportion to its risk.
Change control for AI
A good AI change record shows the reason for the change, the risk tier of the model, the tests run against the baseline (the current approved version), results for accuracy and fairness, approval by the model owner, and the deployment record.
Segregation of duties applies: the person who trains a model should not alone approve and deploy it to production. In automated pipelines, the approval step must be enforced by the pipeline and logged.
Emergency changes, such as disabling a misbehaving model, are acceptable if they are logged, approved after the fact and reviewed.
Champion and challenger releases
A new model (challenger) can be compared with the current model (champion) on the same data, or run in shadow mode alongside it without affecting decisions. Gradual rollouts expose a small share of traffic first. These are good practice and also good evidence: the comparison results show the change was assessed.
Configuration management
| Configuration item | Example |
|---|---|
| Model version and weights file | Version 7, identified by a hash |
| Training dataset version | Snapshot identifier |
| Hyperparameters and decision thresholds | Approval threshold 0.72 |
| Prompts, system instructions and guardrail settings | Prompt template version 3 |
| Feature definitions | List and logic for each input |
| Libraries and runtime | Framework and dependency versions |
| Agent tools and permissions | Allowed tools and scopes |
These should sit in a configuration management database or model registry, with an approved baseline. The auditor compares production with the baseline. An unexplained difference means an unauthorised change or a broken process.
Vendor model changes
Cloud AI services may update or retire model versions. The auditor checks that contracts require notice of changes, that the organization pins a model version where possible, and that it retests before moving to a new version.
Rollback
Every AI change needs a rollback plan: the previous model, data and configuration must be retained and restorable. Retaining only the newest model makes recovery from a bad release slow and risky.
A worked example
A bank's customer chatbot started giving investment suggestions, which it is not permitted to do. The auditor finds no code change in the last month. The prompt template, however, was edited directly in the production console by a product manager to make answers "more helpful".
The auditor reports that prompts were not under change or configuration control. The recommendation is to treat prompts as configuration items, restrict production edit access, and require testing and approval for prompt changes.
How the exam asks about this
- "Which change presents the GREATEST risk if not covered by change management?" Retraining or prompt changes that alter model behaviour without any code change.
- "What is the BEST evidence that a retrained model was properly authorized?" A change record with test results against the baseline and approval by the model owner before deployment.
- "What should the auditor do FIRST to detect unauthorized AI changes?" Compare the production configuration with the approved baseline.
What to remember for the exam
- Retraining, threshold, prompt, retrieval and vendor changes are all changes.
- Test changes against the approved baseline, including fairness.
- Segregate training, approval and deployment duties.
- Keep previous versions for rollback.
- Compare production with the baseline to find unauthorised change.
An AI change process that only looks at code repositories will miss the changes most likely to alter behaviour: data, thresholds and prompts.
Common mistakes
Assuming no code change means no change. Data and configuration changes alter AI behaviour as much as code.
Ignoring vendor updates. A provider can change the underlying model; the organization must detect it and retest.
Allowing direct production edits. Prompts and thresholds edited in a console bypass every gate.
Check yourself
Why is a decision threshold change treated as an AI change?
Because it directly alters outcomes such as approval or alert rates, even though the model itself is unchanged.
What is shadow mode?
Running a new model alongside the current one on live inputs without its outputs affecting decisions, so results can be compared safely.
Why must previous model versions be retained?
So a bad release can be rolled back quickly to a known, approved state.
Topic 3 of 8 in AI Operations
Test yourself on AI Operations
The full ISACA AAIA question bank has 750 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by ISACA. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.