AI Safety, Security, and Governance · Topic 7 of 7
Monitoring, Transparency and Fairness
Monitor FMs for misuse, drift and policy violations, explain answers with agent traces and citations, and test fairness with A/B tests and LLM as a judge.
- 5 min
- Guardrail metrics live in
AWS/Bedrock/Guardrails;InvocationsIntervenedshows how often guardrails acted. - GuardDuty analyzes CloudTrail for suspicious Bedrock activity; AWS Config catches configuration drift.
- EventBridge plus Lambda or Step Functions automates remediation such as rollback.
What it is
Safety controls decay. Users find new ways to misuse an app, source data changes, a new model version answers differently and a guardrail that worked last quarter starts missing things. Continuous monitoring catches this and triggers a response before an auditor or a customer does.
Transparency lets users and reviewers see why the system answered as it did. Fairness evaluation checks that answers do not treat groups of people differently in ways the business cannot justify.
How it works
Continuous monitoring and automated remediation
| Signal | Source | What it shows |
|---|---|---|
| Guardrail interventions | CloudWatch namespace AWS/Bedrock/Guardrails, metric InvocationsIntervened by policy type and input or output | Spikes in blocked content, attacks or PII |
| Model usage and errors | CloudWatch Bedrock runtime metrics | Unusual volume, throttling, cost spikes |
| Prompts and responses | Model invocation logs, decision logs | Content for audit and offline evaluation |
| Suspicious API activity | Amazon GuardDuty analyzing CloudTrail | For example, an unusual identity removing a guardrail |
| Configuration drift | AWS Config rules, CloudTrail | Logging turned off, guardrail detached |
Misuse detection looks for patterns: one user hitting the prompt attack filter repeatedly, a burst of denied topic blocks, or traffic from an unexpected role. A CloudWatch alarm on the guardrail metric or a metric filter on logs starts the response.
Drift detection compares current behavior with a baseline. Run a scheduled evaluation on a fixed test set and on a sample of live traffic, and alarm when grounding, relevance or harmfulness scores move.
Bias drift monitoring: for ML models hosted on Amazon SageMaker AI, SageMaker Model Monitor with SageMaker Clarify tracks bias metrics over time against a baseline. For FM outputs, schedule fairness evaluations (below) and publish the results as CloudWatch custom metrics.
Automated remediation connects alarms to action through Amazon EventBridge: Lambda disables an abusive API key, Step Functions rolls back to the previous prompt or guardrail version, or Amazon SNS pages the on-call team.
Token-level redaction and output policy filters: guardrail sensitive information filters mask individual PII values in the response, and CloudWatch Logs data protection policies mask sensitive values in logs. Response logging then keeps an audit record without exposing raw PII.
Blocked content appears in plain text in model invocation logs. A monitoring design that turns logging on must also lock down and expire those logs.
Transparency: reasoning, confidence and evidence
- Reasoning traces: set
enableTraceonInvokeAgentto get the Bedrock agent trace. It shows the pre-processing, orchestration and post-processing steps, with the rationale, the action group or knowledge base calls (invocation input) and what came back (observation), plus a guardrail trace. - User-facing explanations: show a short summary of the steps or the facts used, not the raw trace. Raw traces can expose prompts and internal data.
- Evidence presentation: display the citations from
RetrieveAndGeneratewith links or document names next to each claim. - Confidence metrics: publish retrieval scores and grounding scores per response to CloudWatch as custom metrics. Low-confidence answers can carry a visible note or go to a human.
Fairness evaluations
Fairness for FMs means similar requests get similarly good answers regardless of attributes such as gender, age, ethnicity or region.
| Method | AWS service | How it works |
|---|---|---|
| Counterfactual prompts | Bedrock evaluation job or Lambda harness | Same prompt with only a demographic attribute changed; compare scores |
| LLM as a judge | Amazon Bedrock model evaluations | Judge model scores responses on built-in metrics including Stereotyping and Harmfulness, or custom metrics |
| A/B testing of prompts | Amazon Bedrock Prompt Management and Amazon Bedrock Flows | Compare prompt variants on the same dataset; the application or a flow Condition node splits traffic between versions |
| Fairness metrics over time | CloudWatch custom metrics and dashboards | Score gap between groups, with alarms on a threshold |
Bedrock Prompt Management stores prompts with variants and versions. Compare variants side by side, then deploy a chosen version. Bedrock Flows (formerly Prompt Flows) links prompts, conditions and other nodes, so a flow can branch to prompt nodes that use different prompt versions; the application decides the split and logs which version answered.
LLM as a judge uses one model to score another model's responses and explain each score. Bring your own responses if the generator model is outside Bedrock. Define a custom metric when the built-in ones do not match your fairness rule.
A/B testing finds the better prompt; fairness testing checks every group is served equally by it. The exam may offer a "best average score" option that ignores the gap between groups.
Common mistakes
Monitoring only errors and latency. A system can be fast and error-free while producing biased or ungrounded answers. Monitor quality and safety scores too.
Showing raw agent traces to end users. Traces can reveal system prompts, tool names and internal data. Summarize them for users and keep raw traces for developers and auditors.
Testing fairness once before launch. Model versions, prompts and data change. Schedule evaluations and alarm on drift.
Manual reaction to alarms. An alert that waits for a person to read email is slow. Wire alarms to EventBridge and automate the first response.
What to remember for the exam
- Guardrail metrics live in
AWS/Bedrock/Guardrails;InvocationsIntervenedshows how often guardrails acted. - GuardDuty analyzes CloudTrail for suspicious Bedrock activity; AWS Config catches configuration drift.
- EventBridge plus Lambda or Step Functions automates remediation such as rollback.
enableTraceonInvokeAgentreturns the agent reasoning trace, including rationale and observations.- Citations and confidence scores make answers transparent; publish scores to CloudWatch.
- Bedrock model evaluations with LLM as a judge score Stereotyping, Harmfulness and custom metrics; Prompt Management variants and Flows support A/B testing.
Check yourself
How can a team see why a Bedrock agent called a specific API?
Enable the trace on InvokeAgent and read the orchestration trace, which includes the rationale, the action group input and the observation.
A new prompt version raises average quality but widens the score gap between two customer groups. What should the fairness process do?
Flag the version and block or roll back the release, because the gap between groups breaks the fairness threshold even if the average improved.
Guardrail interventions on the input suddenly triple. How do you respond automatically?
Alarm on the InvocationsIntervened metric and trigger an EventBridge rule that runs Lambda or Step Functions to throttle the source and notify the team.
Topic 7 of 7 in AI Safety, Security, and Governance
Test yourself on AI Safety, Security, and Governance
The full AWS AIP-C01 question bank has 827 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.