AI Risk Management · Topic 4 of 4
AI Vendor and Supply Chain Management
Managing AI vendors and supply chain: model providers, embedded SaaS AI, open-source models, due diligence, contract terms, monitoring and secure exit.
- 6 min
- Includes a diagram
- The AI supply chain includes model providers, embedded SaaS AI, open-source models, data and libraries.
- Ask about training on your data, retention, underlying models and change notification.
- Put data use, deletion, change and incident notification and flow-down terms in the contract.
What it is
Few organizations build AI entirely in-house. They call model provider APIs, switch on AI features inside SaaS products, download open-source models, buy datasets and labelling services, and rely on AI frameworks and libraries. Each is a link in the AI supply chain.
AI vendor management embeds security requirements into these relationships, verifies them before and after contract, and monitors them as vendors change their models and terms. Outsourcing the work never outsources the accountability.
How it works
Types of AI supplier
| Supplier | Typical risk |
|---|---|
| Foundation model provider (API) | Prompts and data leave the organization; model changes without notice; outage |
| SaaS with embedded AI | AI features enabled by update; data used to improve vendor models; hidden fourth-party model provider |
| Open-source model hub | Unknown provenance; malicious or backdoored model files; licence restrictions |
| Dataset supplier | Poisoned, biased or unlicensed data; personal data without lawful basis |
| Labelling service | Access to raw data; quality and integrity of labels |
| AI frameworks and libraries | Software vulnerabilities in the AI stack |
Fourth parties are common. A SaaS vendor's AI assistant may send customer data to a separate model provider, so due diligence must reach that provider too.
AI-specific due diligence
On top of normal security due diligence, the manager asks:
- Is our data, including prompts and outputs, used to train or improve the vendor's models? Can we opt out?
- How long are prompts and outputs retained, where, and who can access them?
- Which underlying models and subprocessors are used?
- How does the vendor test its AI: red teaming, bias evaluation, safety evaluation? Can we see results?
- Is there documentation such as model cards, an AI BOM, or an independent attestation such as ISO/IEC 42001 certification or an assurance report whose scope covers the AI service?
- How will we be notified of model changes, deprecation and incidents?
- What controls can we configure: logging, content filtering, data residency, access?
- For EU high-risk uses, can the vendor show its conformity as a provider?
Questionnaire answers are verified against independent evidence for higher-risk suppliers, as with any vendor.
Contract terms for AI
| Term | Purpose |
|---|---|
| Data use restrictions | No training on our data without consent; purpose limits |
| Retention and deletion | Defined retention; deletion on request and at exit, with evidence |
| Change notification | Advance notice of model replacement, retirement or material behaviour change |
| Incident notification | Defined time to notify AI incidents and breaches |
| Right to audit or assurance | Access to reports, test results or audits |
| Subprocessor flow-down | Equivalent obligations on model providers and other fourth parties |
| Regulatory support | Information needed for our own compliance and impact assessments |
| Intellectual property and liability | Ownership of outputs; responsibility for infringing outputs |
Embedding and verifying requirements
Security requirements are written into the request for proposal, verified in due diligence, embedded in the contract and then checked in operation. Verification includes confirming configuration settings such as training opt-out, testing the integrated solution in the organization's own context and reviewing updated assurance reports.
Vendors change AI faster than ordinary software. A model update can change behaviour overnight, so monitoring covers model version changes, terms of service changes, new subprocessors and vendor incidents, each a trigger to reassess.
Open-source models and components
| Control | Why |
|---|---|
| Approved sources and a curated internal registry | Teams pull only vetted models |
| Provenance and integrity checks | Verify publisher, hashes or signatures |
| Model scanning | Detect malicious code in model files |
| Safer file formats | Prefer formats that store weights without executable code, such as safetensors |
| Licence review | Some model licences restrict commercial use |
| AI BOM | Record what was used, for later vulnerability response |
Exit and concentration risk
Exit planning covers deletion of data and fine-tuned assets held by the vendor, portability of prompts, evaluation sets and fine-tuning data, and an alternative model or provider. If many business processes depend on one provider, that concentration is reported as a risk in its own right.
A worked example
A collaboration platform announces a new AI assistant, enabled by default, that summarizes chats and documents. The vendor's notice says a third-party model provider processes the content.
The security manager treats this as a supplier change. The assistant is disabled until due diligence confirms data use, retention and the model provider's obligations. The contract is amended for training opt-out and change notification, and the feature is enabled for approved groups.
How the exam asks about this
- "What is MOST important to confirm before staff use a vendor's generative AI service with confidential data?" Whether the vendor uses or retains the data, including for training, and what contractual protections apply.
- "A vendor replaces the model behind a service the organization uses. What should the security manager do?" Treat it as a change: reassess risk and retest before relying on it.
- "Which control BEST reduces the risk of malicious open-source model files?" Use approved sources with integrity checks and model scanning before use.
What to remember for the exam
- The AI supply chain includes model providers, embedded SaaS AI, open-source models, data and libraries.
- Ask about training on your data, retention, underlying models and change notification.
- Put data use, deletion, change and incident notification and flow-down terms in the contract.
- Monitor vendor model and terms changes as reassessment triggers.
- Vet open-source models for provenance, integrity, malicious code and licence.
AI features switched on by a vendor update can bypass procurement entirely. Supplier monitoring and change processes must catch new AI capabilities in existing services.
Common mistakes
Assuming an existing contract covers new AI features. Older terms rarely address training on customer data, model changes or fourth-party model providers.
Trusting a model because it is popular. Download counts are not provenance; verify the source and scan the files.
Forgetting exit. Fine-tuned models, embeddings and logs held by the vendor must be deleted or returned.
Check yourself
Why are fourth parties a particular concern with embedded AI?
A SaaS vendor's AI feature often sends data to a separate model provider, which must also meet the organization's requirements.
Name two AI-specific contract terms.
Any two of no training on customer data, retention and deletion, model change notification, incident notification, subprocessor flow-down or regulatory support.
Why prefer model file formats that store only weights?
Some formats can execute code when loaded, so weight-only formats reduce the risk of malicious model files.
Topic 4 of 4 in AI Risk Management
Test yourself on AI Risk Management
The full ISACA AAISM question bank has 747 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by ISACA. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.