AI Safety, Security, and Governance · Topic 7 of 7

Monitoring, Transparency and Fairness

Monitor FMs for misuse, drift and policy violations, explain answers with agent traces and citations, and test fairness with A/B tests and LLM as a judge.

  • 5 min
In a nutshell
  • Guardrail metrics live in AWS/Bedrock/Guardrails; InvocationsIntervened shows how often guardrails acted.
  • GuardDuty analyzes CloudTrail for suspicious Bedrock activity; AWS Config catches configuration drift.
  • EventBridge plus Lambda or Step Functions automates remediation such as rollback.

What it is

Safety controls decay. Users find new ways to misuse an app, source data changes, a new model version answers differently and a guardrail that worked last quarter starts missing things. Continuous monitoring catches this and triggers a response before an auditor or a customer does.

Transparency lets users and reviewers see why the system answered as it did. Fairness evaluation checks that answers do not treat groups of people differently in ways the business cannot justify.

FM trafficmetrics and logsCloudWatchguardrail metrics, decision logsalarmEventBridgeruleremediateLambda or Step Functionsfix or escalate

How it works

Continuous monitoring and automated remediation

SignalSourceWhat it shows
Guardrail interventionsCloudWatch namespace AWS/Bedrock/Guardrails, metric InvocationsIntervened by policy type and input or outputSpikes in blocked content, attacks or PII
Model usage and errorsCloudWatch Bedrock runtime metricsUnusual volume, throttling, cost spikes
Prompts and responsesModel invocation logs, decision logsContent for audit and offline evaluation
Suspicious API activityAmazon GuardDuty analyzing CloudTrailFor example, an unusual identity removing a guardrail
Configuration driftAWS Config rules, CloudTrailLogging turned off, guardrail detached

Misuse detection looks for patterns: one user hitting the prompt attack filter repeatedly, a burst of denied topic blocks, or traffic from an unexpected role. A CloudWatch alarm on the guardrail metric or a metric filter on logs starts the response.

Drift detection compares current behavior with a baseline. Run a scheduled evaluation on a fixed test set and on a sample of live traffic, and alarm when grounding, relevance or harmfulness scores move.

Bias drift monitoring: for ML models hosted on Amazon SageMaker AI, SageMaker Model Monitor with SageMaker Clarify tracks bias metrics over time against a baseline. For FM outputs, schedule fairness evaluations (below) and publish the results as CloudWatch custom metrics.

Automated remediation connects alarms to action through Amazon EventBridge: Lambda disables an abusive API key, Step Functions rolls back to the previous prompt or guardrail version, or Amazon SNS pages the on-call team.

Token-level redaction and output policy filters: guardrail sensitive information filters mask individual PII values in the response, and CloudWatch Logs data protection policies mask sensitive values in logs. Response logging then keeps an audit record without exposing raw PII.

Watch out

Blocked content appears in plain text in model invocation logs. A monitoring design that turns logging on must also lock down and expire those logs.

Transparency: reasoning, confidence and evidence

  • Reasoning traces: set enableTrace on InvokeAgent to get the Bedrock agent trace. It shows the pre-processing, orchestration and post-processing steps, with the rationale, the action group or knowledge base calls (invocation input) and what came back (observation), plus a guardrail trace.
  • User-facing explanations: show a short summary of the steps or the facts used, not the raw trace. Raw traces can expose prompts and internal data.
  • Evidence presentation: display the citations from RetrieveAndGenerate with links or document names next to each claim.
  • Confidence metrics: publish retrieval scores and grounding scores per response to CloudWatch as custom metrics. Low-confidence answers can carry a visible note or go to a human.
User questioninvoke with traceBedrock agentorchestrationtraceAppsummarize stepsshowUseranswer, sources, confidenceAppcustom metricsCloudWatchconfidence trends

Fairness evaluations

Fairness for FMs means similar requests get similarly good answers regardless of attributes such as gender, age, ethnicity or region.

MethodAWS serviceHow it works
Counterfactual promptsBedrock evaluation job or Lambda harnessSame prompt with only a demographic attribute changed; compare scores
LLM as a judgeAmazon Bedrock model evaluationsJudge model scores responses on built-in metrics including Stereotyping and Harmfulness, or custom metrics
A/B testing of promptsAmazon Bedrock Prompt Management and Amazon Bedrock FlowsCompare prompt variants on the same dataset; the application or a flow Condition node splits traffic between versions
Fairness metrics over timeCloudWatch custom metrics and dashboardsScore gap between groups, with alarms on a threshold

Bedrock Prompt Management stores prompts with variants and versions. Compare variants side by side, then deploy a chosen version. Bedrock Flows (formerly Prompt Flows) links prompts, conditions and other nodes, so a flow can branch to prompt nodes that use different prompt versions; the application decides the split and logs which version answered.

LLM as a judge uses one model to score another model's responses and explain each score. Bring your own responses if the generator model is outside Bedrock. Define a custom metric when the built-in ones do not match your fairness rule.

Exam tip

A/B testing finds the better prompt; fairness testing checks every group is served equally by it. The exam may offer a "best average score" option that ignores the gap between groups.

Common mistakes

Monitoring only errors and latency. A system can be fast and error-free while producing biased or ungrounded answers. Monitor quality and safety scores too.

Showing raw agent traces to end users. Traces can reveal system prompts, tool names and internal data. Summarize them for users and keep raw traces for developers and auditors.

Testing fairness once before launch. Model versions, prompts and data change. Schedule evaluations and alarm on drift.

Manual reaction to alarms. An alert that waits for a person to read email is slow. Wire alarms to EventBridge and automate the first response.

What to remember for the exam

  • Guardrail metrics live in AWS/Bedrock/Guardrails; InvocationsIntervened shows how often guardrails acted.
  • GuardDuty analyzes CloudTrail for suspicious Bedrock activity; AWS Config catches configuration drift.
  • EventBridge plus Lambda or Step Functions automates remediation such as rollback.
  • enableTrace on InvokeAgent returns the agent reasoning trace, including rationale and observations.
  • Citations and confidence scores make answers transparent; publish scores to CloudWatch.
  • Bedrock model evaluations with LLM as a judge score Stereotyping, Harmfulness and custom metrics; Prompt Management variants and Flows support A/B testing.

Check yourself

  • How can a team see why a Bedrock agent called a specific API?

    Enable the trace on InvokeAgent and read the orchestration trace, which includes the rationale, the action group input and the observation.

  • A new prompt version raises average quality but widens the score gap between two customer groups. What should the fairness process do?

    Flag the version and block or roll back the release, because the gap between groups breaks the fairness threshold even if the average improved.

  • Guardrail interventions on the input suddenly triple. How do you respond automatically?

    Alarm on the InvocationsIntervened metric and trigger an EventBridge rule that runs Lambda or Step Functions to throttle the source and notify the team.

Topic 7 of 7 in AI Safety, Security, and Governance

Test yourself on AI Safety, Security, and Governance

The full AWS AIP-C01 question bank has 827 questions, with timed practice exams and an explanation for every answer.

This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.