AI Technologies and Controls · Topic 1 of 7
AI Security Architecture and Design Principles
Designing security architecture for AI: system components, trust boundaries, untrusted model output, least privilege, segregation and threat modelling.
- 6 min
- Includes a diagram
- The model is not a security boundary; enforce controls in the surrounding system.
- Key trust boundaries: user input, external content, model output, tools and training data.
- Treat model output as untrusted before it reaches other systems.
What it is
AI security architecture is the structure of controls around an AI system: where its components sit, how they connect, where trust ends and which control protects each boundary. Good architecture limits the damage when a model is fooled, which matters because models can always be fooled to some degree.
The central design idea is simple: the model is not a security boundary. Instructions in a system prompt, or a model's tendency to refuse, cannot enforce access control. Enforcement happens in the surrounding system.
How it works
Components of a typical AI system
| Component | Role | Key security concern |
|---|---|---|
| Data pipeline | Collects and prepares training and retrieval data | Integrity, provenance, access |
| Training environment | Builds or fine-tunes models | Segregation, access to sensitive data |
| Model registry | Stores approved model versions | Integrity, approval status |
| Inference endpoint | Serves the model | Authentication, rate limits, isolation |
| Orchestration layer | Builds prompts, calls tools, manages context | Separating trusted instructions from untrusted content |
| Retrieval store | Supplies documents or embeddings at query time | Access control, poisoning |
| Tools and integrations | Let the AI read or change other systems | Least privilege, approval |
| User interface | Where people interact | Disclosure, input limits, feedback |
| Logging and monitoring | Records activity | Completeness, privacy of logs |
Trust boundaries
A trust boundary is where data moves between zones of different trust. In AI systems the important boundaries are:
- User input into the model: may contain injection attempts.
- External content into the context: web pages, emails and documents can carry indirect injection.
- Model output into other systems: output is untrusted and must be validated before it reaches a database, browser, shell or API.
- Model to tools: every action the AI can take is a privilege that can be abused.
- Training data into the model: poisoned data becomes model behaviour.
Treating model output like user input is the AI version of a classic rule. Output passed unchecked into a web page or query can create injection flaws in downstream systems.
Design principles
| Principle | How it applies to AI |
|---|---|
| Least privilege | Agents and AI services get only the tools, data and permissions their task needs |
| Defence in depth | Input filtering, output filtering, access control, monitoring and human review together |
| Zero trust | Authenticate every user, service and agent; authorize every request |
| Segregation | Separate training, testing and production; isolate model serving from sensitive back ends |
| Secure by default | Logging on, risky tools off, conservative limits until reviewed |
| Fail safe | When a guardrail or model service fails, the system refuses or falls back rather than passing content through |
| Complete mediation | The application, not the model, checks whether the user may see each piece of retrieved data |
Identities for AI
AI services and agents are non-human identities. Each needs its own credentials, managed through the identity platform, with scoped permissions, rotation and monitoring. Where an agent acts for a user, it should act with that user's permissions, not a broad service account, so the AI can never reach more than the user could.
Secrets such as API keys must never be placed in prompts or system prompts, which can leak. They belong in a secrets manager used by the application.
Threat modelling AI systems
Threat modelling is done at design time and repeated at major changes. The team draws the data flows and trust boundaries, then asks what could go wrong at each, using sources such as MITRE ATLAS and the OWASP Top 10 for LLM Applications alongside the organization's usual method. The output is a list of threats with the control that addresses each.
Protecting the model itself
Model weights are valuable intellectual property and their integrity determines behaviour. Controls include access control and encryption for weights at rest, integrity checks such as hashes or signatures on deployment, approval gates in the registry, and rate limits and monitoring on the inference endpoint to make extraction harder.
A worked example
A team designs an assistant that answers staff questions from the document management system. The first design gives the assistant a service account that can read every document.
The security manager rejects this. Retrieval must run with the requesting user's permissions, so the assistant can only quote documents the user is already allowed to open. Output is checked for sensitive data before display, logs capture queries and sources, and the assistant has no write access to anything.
How the exam asks about this
- "Which is the BEST way to prevent an AI assistant from revealing documents a user is not authorized to see?" Enforce the user's access rights in retrieval, outside the model.
- "How should output from a language model be treated before it is passed to another application?" As untrusted input that must be validated or encoded.
- "What is the MOST important principle when giving an AI agent access to tools?" Least privilege.
What to remember for the exam
- The model is not a security boundary; enforce controls in the surrounding system.
- Key trust boundaries: user input, external content, model output, tools and training data.
- Treat model output as untrusted before it reaches other systems.
- Agents are non-human identities with least privilege, ideally acting with the user's own rights.
- Threat model AI designs at the start and at major changes.
A system prompt that says "never reveal confidential data" is not a control. If the model can reach data, assume an attacker can eventually make it reveal that data.
Common mistakes
Relying on prompt instructions for security. Prompts guide behaviour but can be overridden; access control must sit outside the model.
Giving an AI one powerful service account. That lets any user reach everything the account can reach through the AI.
Trusting output because the model is internal. Output can still carry injected content or errors into downstream systems.
Check yourself
Why is the model not a security boundary?
Its behaviour can be manipulated by inputs, so instructions to the model cannot reliably enforce access control or other security rules.
What is a key trust boundary when an AI reads emails or web pages?
External content entering the model's context, because it can carry indirect prompt injection.
Where should secrets such as API keys used by an AI application be stored?
In a secrets manager used by the application, never in prompts or system prompts.
Topic 1 of 7 in AI Technologies and Controls
Test yourself on AI Technologies and Controls
The full ISACA AAISM question bank has 747 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by ISACA. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.