Operational Excellence and Automation · Topic 7 of 7
Evaluating and Monitoring AI Workloads
Validate model and RAG outputs with Amazon Bedrock Evaluations, and monitor latency, tokens, agents and model drift with CloudWatch and SageMaker AI.
- 5 min
- Bedrock Evaluations supports programmatic, LLM as a judge, human and RAG evaluations.
- Use evaluations as a release gate whenever the model, prompt or knowledge base changes.
- Guardrails act at run time; evaluations judge quality before release.
What it is
AI workloads need the same operational discipline as any other application, plus two extra checks. Before release you must show that the model, prompt or retrieval setup gives good answers. After release you must watch quality as well as speed, because a model can stay fast while its answers get worse.
For the exam, know which AWS feature evaluates outputs before deployment and which one monitors inference, agents and drift in production.
How it works
Evaluating outputs before release
Amazon Bedrock Evaluations scores models and knowledge bases against a dataset, and can also evaluate models and Retrieval Augmented Generation (RAG) sources outside Bedrock.
| Evaluation type | How it scores | Use it when |
|---|---|---|
| Programmatic (automatic) | Computed metrics on a built-in or custom prompt dataset | Quick comparison of models on a task |
| LLM as a judge | A second model scores each response and explains why | Quality criteria such as correctness or helpfulness at scale |
| Human-based | Your employees or subject experts rate responses | Subjective or high-stakes judgments |
| RAG evaluation | Scores retrieval relevance and generated response quality against ground truth | Tuning an Amazon Bedrock Knowledge Base or other RAG source |
Results land in S3 and in the console as a report. Treat a passing evaluation as a release gate: run the job when a prompt, model version or knowledge base changes, and compare scores against the current production setup.
Evaluation checks quality before release. Amazon Bedrock Guardrails filter content, deny topics and check grounding at run time. They complement each other and do not replace each other.
Monitoring inference performance
| Workload | Metrics to watch | Where |
|---|---|---|
| Amazon Bedrock models | Invocations, invocation latency, input and output token counts, throttles, errors | CloudWatch AWS/Bedrock namespace |
| Bedrock request content | Prompts and responses for audit and analysis | Model invocation logging to CloudWatch Logs or S3 |
| SageMaker AI endpoints | Invocations, model latency, overhead latency, 4XX and 5XX errors, instance utilization | CloudWatch |
| AI agents on AgentCore | Session count, latency, duration, token usage, error rates, traces of each step | AgentCore Observability in CloudWatch |
Alarm on latency percentiles and throttles, not only averages, because a few slow generations hurt user experience. Rising token counts per request drive cost and latency, so watch them as an early warning.
Amazon Bedrock AgentCore Observability emits telemetry in an OpenTelemetry compatible format and stores metrics, spans and logs in CloudWatch. Its trace view shows each step of an agent run, so you can see which tool call or model call failed or slowed down. Agents built elsewhere can be instrumented with OpenTelemetry to send the same kind of data.
Monitoring model quality and drift
A deployed model can degrade when real data stops looking like the training data. Amazon SageMaker Model Monitor compares live traffic to a baseline.
| Monitor type | Detects |
|---|---|
| Data quality | Changes in input feature statistics compared with training data |
| Model quality | Falling accuracy or similar metrics, once ground truth labels arrive |
| Bias drift | Changes in bias metrics, using SageMaker Clarify |
| Feature attribution drift | Changes in which features drive predictions |
Model Monitor needs data capture turned on for the endpoint and a baseline built from training data. Schedules run the checks and publish results to CloudWatch, where an alarm can start a retraining pipeline.
Model quality monitoring needs ground truth labels, which often arrive late. For generative models with no simple label, use periodic evaluation jobs on samples of real prompts as the quality signal.
How the exam asks about this
- "Compare three foundation models on the company's own support questions for correctness before choosing one, with minimal human effort" -> a Bedrock Evaluations job that uses an LLM as a judge on a custom prompt dataset.
- "A fraud model's accuracy fell over months as customer behavior changed" -> SageMaker Model Monitor with data capture and data quality or model quality monitoring, alarming to trigger retraining.
- "Find which step of a multi-tool AI agent causes slow responses in production" -> AgentCore Observability traces in CloudWatch.
What to remember for the exam
- Bedrock Evaluations supports programmatic, LLM as a judge, human and RAG evaluations.
- Use evaluations as a release gate whenever the model, prompt or knowledge base changes.
- Guardrails act at run time; evaluations judge quality before release.
- Bedrock publishes invocation, latency, token and throttle metrics to CloudWatch; invocation logging captures prompts and responses.
- AgentCore Observability gives agent traces and metrics in CloudWatch in an OpenTelemetry compatible format.
- SageMaker Model Monitor detects data, model quality, bias and feature attribution drift against a baseline.
Latency and error metrics can look healthy while answers are wrong. When a question mentions declining accuracy or relevance, the answer is evaluation or drift monitoring, not a CloudWatch latency alarm.
Common mistakes
Treating Guardrails as quality testing. Guardrails block harmful or off-topic content. They do not measure whether answers are correct, which is the job of evaluations.
Monitoring drift without a baseline. Model Monitor compares against baseline statistics. Without a baseline and data capture there is nothing to compare.
Watching only average latency. Generative responses vary in length. Percentile latency and token counts reveal the slow tail that users feel.
Check yourself
Which service compares candidate models on a company's own prompts before release?
Amazon Bedrock Evaluations, using a custom prompt dataset with programmatic, judge model or human scoring.
What must be enabled before SageMaker Model Monitor can detect data drift on an endpoint?
Data capture on the endpoint and a baseline created from the training data.
Where do you see the step-by-step trace of an agent running on Amazon Bedrock AgentCore?
In AgentCore Observability, which stores its traces and metrics in Amazon CloudWatch.
Topic 7 of 7 in Operational Excellence and Automation
Test yourself on Operational Excellence and Automation
The full AWS SAP-C03 question bank has 850 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.