Operational Efficiency and Optimization for GenAI Applications · Topic 6 of 7
GenAI Observability and Monitoring
Monitor FM apps with Bedrock CloudWatch metrics, model invocation logs, tracing, anomaly detection on tokens and cost, dashboards and audit trails.
- 5 min
- Key Bedrock metrics: Invocations, InvocationLatency, TimeToFirstToken, InputTokenCount, OutputTokenCount, InvocationThrottles, client and server errors.
- Invocation logging: off by default, S3 and or CloudWatch Logs, same account and Region, large bodies to S3.
- Traces with X-Ray or OpenTelemetry show per-step latency across retrieval, guardrails, models and tools.
What it is
Traditional monitoring asks "is it up and fast". GenAI monitoring also asks "is it right, is it safe, and what does it cost". A model can return HTTP 200 with a confident, wrong answer, so status codes alone tell you little.
Holistic observability combines four layers: operational metrics, request and response content, traces across every step, and business impact. Each layer answers a different question.
How it works
Operational metrics
Amazon Bedrock publishes runtime metrics to CloudWatch in the AWS/Bedrock namespace, with and without the ModelId dimension.
| Metric | Use it to watch |
|---|---|
Invocations | Request volume |
InvocationLatency, TimeToFirstToken | Total latency and streaming responsiveness |
InputTokenCount, OutputTokenCount | Token usage and cost drivers |
InvocationThrottles | Quota pressure |
InvocationClientErrors, InvocationServerErrors | Bad requests versus service-side errors |
CacheReadInputTokenCount, CacheWriteInputTokenCount | Prompt cache effectiveness |
Bedrock Guardrails, Bedrock Agents and AgentCore publish their own metrics too. Requests that fail before Bedrock knows the model are published without ModelId, so filtering by model can undercount errors.
Model invocation logs
Model invocation logging captures the full request, response and metadata for Converse, ConverseStream, InvokeModel and InvokeModelWithResponseStream calls.
- It is disabled by default; turn it on per account and Region in the Bedrock settings or with
PutModelInvocationLoggingConfiguration. - Destinations are Amazon S3, CloudWatch Logs or both, in the same account and Region.
- Bodies up to 100 KB are inline; larger bodies and binary data go to S3 as separate objects.
- Each entry includes the caller's
identity.arn, the model ID, token counts and optional caller-suppliedrequestMetadatatags.
Query them with CloudWatch Logs Insights for fast troubleshooting, or with Amazon Athena over S3 for long-term analysis, such as token use by principal or by team tag.
Invocation logs contain full prompts and responses, which may include personal data. Encrypt the destination with AWS KMS, restrict access, set retention, and consider CloudWatch Logs data protection policies to mask sensitive fields.
Tracing FM interactions
A single user request may involve a guardrail check, retrieval, several model calls and tool calls. Performance tracing with AWS X-Ray or OpenTelemetry (for example, AWS Distro for OpenTelemetry) records each step as a span, so you see which step is slow or failing. Agents on AgentCore emit OpenTelemetry-compatible traces viewable in CloudWatch.
Pass a correlation ID through every step and include it in requestMetadata, so traces, invocation logs and application logs can be joined during an investigation.
GenAI-specific KPIs
| KPI | How to measure it |
|---|---|
| Token usage and cost per request, user or tenant | Token metrics and invocation logs, priced per model |
| Prompt effectiveness | Task success rate, user feedback, retries per session, by prompt version |
| Hallucination rate | Sampled responses scored by an LLM-as-a-judge evaluation or guardrail contextual grounding checks, published as a custom metric |
| Response quality | Evaluation scores on sampled traffic and on a fixed test set |
| Response drift | Trend of judge scores, refusal rate and output length after model or prompt changes |
| Guardrail interventions | Guardrails metrics: rising blocks may mean abuse or an overly strict policy |
Publish custom metrics with PutMetricData or the CloudWatch embedded metric format from your application.
Anomaly detection
- CloudWatch anomaly detection builds an expected band for a metric. Alarm on
InputTokenCountorOutputTokenCountoutside the band to catch token bursts, such as a prompt loop or abuse. - Apply anomaly detection to quality metrics, such as judge score or average output length, to catch response drift.
- AWS Cost Anomaly Detection flags unusual spend at the service or cost allocation tag level; AWS Budgets alerts on thresholds.
- For per-team or per-application cost, invoke through application inference profiles with cost allocation tags.
Business impact and integrated dashboards
Operators and business owners need different views of the same data.
- Operational dashboards: latency, errors, throttles and tokens per model.
- Business impact: cases deflected, conversion, time saved and cost per resolved request, from custom metrics.
- User interaction tracking: sessions, feedback ratings, abandoned conversations.
- Model behavior patterns: output length, refusal rate and topics over time.
Compliance and forensic traceability
- AWS CloudTrail records Bedrock API activity: who called which API, and when.
- Model invocation logs record what was asked and answered.
- Store audit logs in S3 with versioning and S3 Object Lock for tamper-resistant retention.
- AWS Config checks that configurations, such as logging enabled or encryption on, stay compliant.
How the exam asks about this
- "Investigate exactly what the model returned to a customer last week" -> model invocation logs in S3 queried with Athena.
- "Alert when token usage suddenly spikes outside normal patterns" -> CloudWatch anomaly detection alarm on token count metrics.
- "Show executives the cost per resolved support case" -> custom business metrics on a CloudWatch dashboard.
Common mistakes
Assuming Bedrock logs prompts automatically. Model invocation logging is off until you configure it.
Relying on CloudTrail for prompt content. CloudTrail is for API activity; prompts and responses are in invocation logs.
Monitoring only latency and errors. Hallucinations and drift return successful responses, so quality KPIs need their own metrics.
What to remember for the exam
- Key Bedrock metrics: Invocations, InvocationLatency, TimeToFirstToken, InputTokenCount, OutputTokenCount, InvocationThrottles, client and server errors.
- Invocation logging: off by default, S3 and or CloudWatch Logs, same account and Region, large bodies to S3.
- Traces with X-Ray or OpenTelemetry show per-step latency across retrieval, guardrails, models and tools.
- Anomaly detection catches token bursts and drift; Cost Anomaly Detection catches spend spikes.
- CloudTrail answers who and when; invocation logs answer what was said.
Check yourself
Where do you find the exact prompt and response for a past Bedrock call?
In model invocation logs, if logging was enabled, delivered to S3 or CloudWatch Logs.
How can you detect a sudden runaway loop that multiplies token use?
Put a CloudWatch anomaly detection alarm on the token count metrics so values outside the expected band raise an alert.
Why are quality KPIs needed in addition to error metrics?
A hallucinated answer is still a successful call, so only evaluation scores and grounding checks reveal it.
Topic 6 of 7 in Operational Efficiency and Optimization for GenAI Applications
Test yourself on Operational Efficiency and Optimization for GenAI Applications
The full AWS AIP-C01 question bank has 827 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.