Operational Efficiency and Optimization for GenAI Applications · Topic 7 of 7

Tool Tracing and GenAI Troubleshooting

Track agent tool calls and multi-agent coordination with traces and baselines, and find hallucinations with golden datasets and output diffing.

  • 6 min
In a nutshell
  • Bedrock Agents trace: enableTrace on InvokeAgent; rationale, invocation input and observation per step.
  • AgentCore Observability: OpenTelemetry-compatible traces and metrics in CloudWatch, including tool calls.
  • Usage baselines plus anomaly detection catch tool loops, wrong tool choice and slow dependencies.

What it is

Agents call tools: Lambda functions, APIs, databases and other agents. A tool performance framework tracks which tools are called, how often, how fast and how often they fail, so you can spot loops, wrong tool choices and slow dependencies.

GenAI troubleshooting deals with failures that traditional ML systems do not have: hallucinated facts, answers that change between identical runs, broken reasoning chains and silent quality loss after a prompt or model change. These need their own tests and evidence.

UserrequestAgentBedrock Agents or AgentCoretool callToolLambda or APIresultAgentanswerUserAgenttraceObservabilityrationale, tool input, observationbaseline checkAnomaly alarm

How it works

Tool call observability

Amazon Bedrock Agents returns a trace when you call InvokeAgent with enableTrace set to true. The trace shows each step of orchestration: the model's rationale, the action group or knowledge base it chose (the invocation input), and the observation it got back. Other trace types cover preprocessing, postprocessing, guardrails and failures.

Amazon Bedrock AgentCore Observability emits OpenTelemetry-compatible traces, spans and metrics into CloudWatch. It tracks session count, latency, duration, token usage and error rates, and shows each step of the agent workflow, including tool calls through AgentCore Gateway.

The tools themselves have normal service metrics. For Lambda-based tools, watch Duration, Errors and Throttles; for APIs, latency and 4XX or 5XX rates.

Call pattern tracking and usage baselines

SignalNormal patternProblem it reveals
Tool calls per sessionStable small rangeA loop: the agent keeps calling the same tool
Distribution of tools chosenMatches request mixWrong tool selection after a description or prompt change
Tool error rateLow and steadyBad parameters generated by the model, or a failing backend
Tool latency at p99SteadyA slow dependency that stretches the whole agent response
Tokens per sessionStable bandRunaway reasoning or very large tool outputs fed back to the model

Build these as custom metrics from traces, then set CloudWatch anomaly detection on them. The learned band is your usage baseline; a break from it raises an alarm before users complain.

Exam tip

When an agent "sometimes calls the wrong tool", the fix is usually clearer tool names, descriptions and parameter schemas, confirmed by trace data. Do not jump to fine tuning.

Multi-agent coordination tracking

With multi-agent collaboration, a supervisor agent routes work to collaborator agents. Trace the whole chain with one session or correlation ID, so you can see which collaborator handled each step, how long each handoff took, and where a request stalled or bounced between agents.

GenAI-specific failure modes

Failure modeSymptomDetection
HallucinationConfident, unsupported factsGolden dataset checks, contextual grounding checks, faithfulness scores
InconsistencyDifferent answers to the same inputOutput diffing across repeated runs
Reasoning errorRight facts, wrong conclusion or stepReasoning path tracing in agent traces
Retrieval missAnswer ignores the relevant documentRetrieve-only evaluation of the knowledge base
Context truncationInstructions or documents silently ignoredToken counts near the context limit in invocation logs
Regression after changeQuality drop after new prompt or model versionGolden dataset run before and after the change

Golden datasets

A golden dataset is a curated set of prompts with verified reference answers and, for RAG, the source passages that should be retrieved. Run it on every prompt, model or retrieval change, and on a schedule to catch drift.

  • Amazon Bedrock Evaluations can run model evaluation jobs and RAG evaluation jobs against your own dataset, with LLM-as-a-judge metrics such as correctness and faithfulness.
  • A drop in score on the golden set is a regression signal even when no user has complained.

Output diffing for consistency

Output diffing compares responses to detect inconsistency or drift.

Golden promptrun N timesModelsame settingsresponsesDiff stepsemantic similaritylow agreementFlag for reviewGolden promptold versionResponse AcompareDiff stepGolden promptnew versionResponse BcompareDiff step

Compare by meaning, not by exact text: use embedding similarity or a judge model, plus exact checks on structured fields such as JSON keys or numbers. High variation on factual prompts often means temperature is too high or the prompt is ambiguous.

Reasoning path tracing

When the final answer is wrong, the trace shows where the logic broke: a wrong tool chosen, a misread tool result, or a bad assumption in the model's rationale. Bedrock Agents traces and AgentCore spans give this step-by-step view; for custom orchestration, log each step's input and output with the correlation ID.

Specialized observability pipelines

Build a pipeline just for GenAI evidence:

  1. Model invocation logs and agent traces land in S3.
  2. AWS Glue catalogs them; Amazon Athena queries them.
  3. A scheduled job samples responses and scores them with an evaluation model.
  4. Scores become CloudWatch custom metrics with alarms and dashboards.

How the exam asks about this

  • "An agent occasionally loops calling the same API until timeout" -> track tool calls per session from traces and alarm on deviations from the baseline.
  • "Detect hallucinations automatically before a new prompt version goes live" -> run a golden dataset evaluation and compare scores with the current version.
  • "Find which step of an agent's plan produced the wrong answer" -> enable the agent trace and inspect rationale, invocation input and observation.

Common mistakes

Debugging agents from the final answer only. Without traces you cannot see which tool or reasoning step failed. Enable traces in every environment.

Exact string comparison for consistency checks. LLM outputs vary in wording even when they agree. Diff by meaning and check structured fields exactly.

Treating hallucination as a code bug. Retries and exception handling do not catch it. It needs grounding checks and evaluation against known answers.

What to remember for the exam

  • Bedrock Agents trace: enableTrace on InvokeAgent; rationale, invocation input and observation per step.
  • AgentCore Observability: OpenTelemetry-compatible traces and metrics in CloudWatch, including tool calls.
  • Usage baselines plus anomaly detection catch tool loops, wrong tool choice and slow dependencies.
  • Golden datasets with Bedrock Evaluations detect hallucinations and regressions.
  • Output diffing checks consistency by meaning across runs and versions.
  • Reasoning path tracing locates the exact step where an agent went wrong.

Check yourself

  • How do you see why a Bedrock agent chose a particular action group?

    Call InvokeAgent with enableTrace set to true and read the orchestration trace's rationale and invocation input.

  • What is a golden dataset used for in troubleshooting?

    It is a curated set of prompts with verified answers that you run after every change to detect hallucinations and regressions.

  • Why diff outputs by semantic similarity instead of exact text?

    Correct LLM answers vary in wording, so exact comparison flags false differences while similarity measures real disagreement.

Topic 7 of 7 in Operational Efficiency and Optimization for GenAI Applications

Test yourself on Operational Efficiency and Optimization for GenAI Applications

The full AWS AIP-C01 question bank has 827 questions, with timed practice exams and an explanation for every answer.

This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.