Operational Efficiency and Optimization for GenAI Applications · Topic 7 of 7
Tool Tracing and GenAI Troubleshooting
Track agent tool calls and multi-agent coordination with traces and baselines, and find hallucinations with golden datasets and output diffing.
- 6 min
- Bedrock Agents trace:
enableTraceon InvokeAgent; rationale, invocation input and observation per step. - AgentCore Observability: OpenTelemetry-compatible traces and metrics in CloudWatch, including tool calls.
- Usage baselines plus anomaly detection catch tool loops, wrong tool choice and slow dependencies.
What it is
Agents call tools: Lambda functions, APIs, databases and other agents. A tool performance framework tracks which tools are called, how often, how fast and how often they fail, so you can spot loops, wrong tool choices and slow dependencies.
GenAI troubleshooting deals with failures that traditional ML systems do not have: hallucinated facts, answers that change between identical runs, broken reasoning chains and silent quality loss after a prompt or model change. These need their own tests and evidence.
How it works
Tool call observability
Amazon Bedrock Agents returns a trace when you call InvokeAgent with enableTrace set to true. The trace shows each step of orchestration: the model's rationale, the action group or knowledge base it chose (the invocation input), and the observation it got back. Other trace types cover preprocessing, postprocessing, guardrails and failures.
Amazon Bedrock AgentCore Observability emits OpenTelemetry-compatible traces, spans and metrics into CloudWatch. It tracks session count, latency, duration, token usage and error rates, and shows each step of the agent workflow, including tool calls through AgentCore Gateway.
The tools themselves have normal service metrics. For Lambda-based tools, watch Duration, Errors and Throttles; for APIs, latency and 4XX or 5XX rates.
Call pattern tracking and usage baselines
| Signal | Normal pattern | Problem it reveals |
|---|---|---|
| Tool calls per session | Stable small range | A loop: the agent keeps calling the same tool |
| Distribution of tools chosen | Matches request mix | Wrong tool selection after a description or prompt change |
| Tool error rate | Low and steady | Bad parameters generated by the model, or a failing backend |
| Tool latency at p99 | Steady | A slow dependency that stretches the whole agent response |
| Tokens per session | Stable band | Runaway reasoning or very large tool outputs fed back to the model |
Build these as custom metrics from traces, then set CloudWatch anomaly detection on them. The learned band is your usage baseline; a break from it raises an alarm before users complain.
When an agent "sometimes calls the wrong tool", the fix is usually clearer tool names, descriptions and parameter schemas, confirmed by trace data. Do not jump to fine tuning.
Multi-agent coordination tracking
With multi-agent collaboration, a supervisor agent routes work to collaborator agents. Trace the whole chain with one session or correlation ID, so you can see which collaborator handled each step, how long each handoff took, and where a request stalled or bounced between agents.
GenAI-specific failure modes
| Failure mode | Symptom | Detection |
|---|---|---|
| Hallucination | Confident, unsupported facts | Golden dataset checks, contextual grounding checks, faithfulness scores |
| Inconsistency | Different answers to the same input | Output diffing across repeated runs |
| Reasoning error | Right facts, wrong conclusion or step | Reasoning path tracing in agent traces |
| Retrieval miss | Answer ignores the relevant document | Retrieve-only evaluation of the knowledge base |
| Context truncation | Instructions or documents silently ignored | Token counts near the context limit in invocation logs |
| Regression after change | Quality drop after new prompt or model version | Golden dataset run before and after the change |
Golden datasets
A golden dataset is a curated set of prompts with verified reference answers and, for RAG, the source passages that should be retrieved. Run it on every prompt, model or retrieval change, and on a schedule to catch drift.
- Amazon Bedrock Evaluations can run model evaluation jobs and RAG evaluation jobs against your own dataset, with LLM-as-a-judge metrics such as correctness and faithfulness.
- A drop in score on the golden set is a regression signal even when no user has complained.
Output diffing for consistency
Output diffing compares responses to detect inconsistency or drift.
Compare by meaning, not by exact text: use embedding similarity or a judge model, plus exact checks on structured fields such as JSON keys or numbers. High variation on factual prompts often means temperature is too high or the prompt is ambiguous.
Reasoning path tracing
When the final answer is wrong, the trace shows where the logic broke: a wrong tool chosen, a misread tool result, or a bad assumption in the model's rationale. Bedrock Agents traces and AgentCore spans give this step-by-step view; for custom orchestration, log each step's input and output with the correlation ID.
Specialized observability pipelines
Build a pipeline just for GenAI evidence:
- Model invocation logs and agent traces land in S3.
- AWS Glue catalogs them; Amazon Athena queries them.
- A scheduled job samples responses and scores them with an evaluation model.
- Scores become CloudWatch custom metrics with alarms and dashboards.
How the exam asks about this
- "An agent occasionally loops calling the same API until timeout" -> track tool calls per session from traces and alarm on deviations from the baseline.
- "Detect hallucinations automatically before a new prompt version goes live" -> run a golden dataset evaluation and compare scores with the current version.
- "Find which step of an agent's plan produced the wrong answer" -> enable the agent trace and inspect rationale, invocation input and observation.
Common mistakes
Debugging agents from the final answer only. Without traces you cannot see which tool or reasoning step failed. Enable traces in every environment.
Exact string comparison for consistency checks. LLM outputs vary in wording even when they agree. Diff by meaning and check structured fields exactly.
Treating hallucination as a code bug. Retries and exception handling do not catch it. It needs grounding checks and evaluation against known answers.
What to remember for the exam
- Bedrock Agents trace:
enableTraceon InvokeAgent; rationale, invocation input and observation per step. - AgentCore Observability: OpenTelemetry-compatible traces and metrics in CloudWatch, including tool calls.
- Usage baselines plus anomaly detection catch tool loops, wrong tool choice and slow dependencies.
- Golden datasets with Bedrock Evaluations detect hallucinations and regressions.
- Output diffing checks consistency by meaning across runs and versions.
- Reasoning path tracing locates the exact step where an agent went wrong.
Check yourself
How do you see why a Bedrock agent chose a particular action group?
Call InvokeAgent with enableTrace set to true and read the orchestration trace's rationale and invocation input.
What is a golden dataset used for in troubleshooting?
It is a curated set of prompts with verified answers that you run after every change to detect hallucinations and regressions.
Why diff outputs by semantic similarity instead of exact text?
Correct LLM answers vary in wording, so exact comparison flags false differences while similarity measures real disagreement.
Topic 7 of 7 in Operational Efficiency and Optimization for GenAI Applications
Test yourself on Operational Efficiency and Optimization for GenAI Applications
The full AWS AIP-C01 question bank has 827 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.