Securing AI Systems · Topic 2 of 7
Model Guardrails and AI Gateway Controls
Implement AI security controls: model evaluation, guardrails, prompt templates, prompt firewalls, rate, token, quota and modality limits, guardrail tests.
- 5 min
- Includes a diagram
- Model controls are evaluation, guardrails and prompt templates; gateway controls sit in front of the model.
- A prompt firewall inspects prompts for injection, jailbreaks and sensitive data before they reach the model.
- Rate limits cap requests; token limits cap size and cost; input quotas cap data size and quantity.
What it is
Security controls for artificial intelligence (AI) systems sit in two places. Model controls shape how the model itself behaves, and gateway controls sit in front of the model and filter or limit every request before it arrives.
An AI gateway is a proxy between users or applications and one or more models. It is the natural place to enforce policy, because every prompt and response passes through it whichever model is behind.
How it works
Model controls
| Control | What it does | Example |
|---|---|---|
| Model evaluation | Tests the model for accuracy, safety, bias and attack resistance before and after release | A test set of harmful requests the model must refuse |
| Model guardrails | Rules and classifiers that block or rewrite unsafe input and output | Refuse instructions for weapons; strip personal data from answers |
| Prompt templates | A fixed prompt structure that places user input in a defined slot, with the system rules around it | User text inserted between markers and labelled as untrusted |
Guardrails can work on the input, the output or both. Output guardrails matter as much as input ones, because a harmless-looking prompt can still lead to a harmful or leaking answer.
Gateway controls
| Control | Purpose | Stops |
|---|---|---|
| Prompt firewall | Inspects prompts, and often responses, for injection patterns, jailbreak phrasing, secrets or sensitive data | Prompt injection, data leakage |
| Rate limits | Caps requests per user, key or address per time period | Abuse, scraping, model theft by mass querying |
| Token limits | Caps tokens per prompt, per response or per period | Oversized prompts, runaway cost, model denial of service (DoS) |
| Input quotas: data size | Caps the size of each upload or request body | Huge files that exhaust processing |
| Input quotas: quantity | Caps the number of files, records or requests in a period | Bulk abuse and extraction |
| Modality limits | Allows only the input types the use case needs, such as text only | Hidden instructions in images or audio, larger attack surface |
| Endpoint access controls | Requires authentication and authorisation to reach the model endpoint, and limits network exposure | Anonymous use, use by unapproved apps |
A token is a word or part of a word, and models charge and compute per token. Token limits are therefore both a security control and a cost control.
A guardrail policy in pseudo-config
# illustrative policy, not a real product syntax
gateway:
auth: required # only approved apps and signed-in users
rate_limit: 60 per minute per user
max_input_tokens: 4000
max_output_tokens: 1000
allowed_modalities: [text]
max_upload_size: 5 MB
prompt_firewall:
block: [injection_patterns, secrets, card_numbers]
output_guardrails:
redact: [personal_data]
block_categories: [malware_code, self_harm]
on_violation: block_and_logThe policy combines identity, limits, input inspection and output inspection. Each line maps to one requirement, which is how exam scenarios are written.
Guardrail testing and validation
Guardrails must be tested like any other control. Testing proves a guardrail blocks what it should; validation confirms it does so in the real deployment without blocking legitimate work.
- Build a test set of known injection, jailbreak and data leak attempts, plus normal requests.
- Run it automatically before every model, prompt or guardrail change, as a regression test.
- Red team the system with people trying new techniques.
- Measure both missed attacks (false negatives) and blocked normal use (false positives).
- Retest after model updates, because a new model version can respond differently to the same guardrail.
A worked example
A legal team wants a contract summariser. The requirements are: only staff may use it, it must accept text and Portable Document Format (PDF) files but not images, a single user must not run up large costs, and summaries must never include client bank details.
The engineer puts the model behind a gateway with single sign-on (SSO), a modality limit of text and PDF files, per-user rate and token limits, and an output guardrail that redacts account numbers. A test set of contracts with planted bank details runs before every release.
How the exam asks about this
- "Users are uploading images that contain hidden instructions to a text-only support bot. Which of the following controls best addresses this?" A modality limit that allows text input only.
- "A single application key sends thousands of near-identical queries per hour to a model's endpoint. Which of the following should be implemented first?" Rate limits per key at the gateway.
- "After a model upgrade, the security team must confirm existing protections still block known jailbreaks. Which of the following should it perform?" Guardrail testing and validation with a regression test set.
What to remember for the exam
- Model controls are evaluation, guardrails and prompt templates; gateway controls sit in front of the model.
- A prompt firewall inspects prompts for injection, jailbreaks and sensitive data before they reach the model.
- Rate limits cap requests; token limits cap size and cost; input quotas cap data size and quantity.
- Modality limits restrict input types to what the use case needs.
- Guardrails need automated regression tests and red teaming, repeated after every model change.
Many guardrails are probabilistic, and all can be bypassed. Never rely on a guardrail alone where a mistake would be serious; back it with least privilege, output checks and human approval.
Common mistakes
Inspecting only the prompt. Many leaks and harmful outputs come from innocent-looking prompts. Inspect responses as well.
Confusing rate limits with token limits. A rate limit counts requests; a token limit counts the size of each request or response. A few huge prompts can pass a rate limit and still exhaust the model.
Testing guardrails once at launch. Model updates, prompt changes and new attack techniques all change results, so testing must be continuous.
Check yourself
What is the main benefit of placing controls in an AI gateway?
Every request and response passes through one point, so policy is enforced consistently whatever model is behind it.
Which control limits the types of input, such as text, images or audio, that a model accepts?
A modality limit.
Why should guardrail tests measure false positives as well as missed attacks?
A guardrail that blocks normal requests harms the business and encourages users to bypass it.
Topic 2 of 7 in Securing AI Systems
Test yourself on Securing AI Systems
The full CompTIA SecAI+ CY0-001 question bank has 654 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by CompTIA. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.