Testing, Validation, and Troubleshooting · Topic 2 of 6

Human Feedback, LLM-as-a-Judge and Agent Evaluation

Collect user ratings and annotations, score outputs with LLM-as-a-Judge and RAG evaluation, and measure agent goal success and tool use with AgentCore.

  • 5 min
In a nutshell
  • Combine user feedback, LLM judges and human annotation for a multi perspective view.
  • SageMaker Ground Truth labels data; Amazon A2I adds human review loops; Bedrock human evaluation jobs compare up to two models with your own team.
  • Bedrock RAG evaluation: retrieve only (context relevance, context coverage) or retrieve and generate (adds faithfulness, citation precision and coverage, correctness and more).

What it is

No single evaluator sees everything. Automated metrics are fast but shallow, judge models scale but can be biased, and humans are accurate but slow and costly.

A thorough assessment combines three perspectives: users who rate real answers, LLM judges that score every response against a rubric, and expert reviewers who annotate samples. Agents add a fourth layer, because you must judge the steps and tool calls, not only the final answer.

Live answersproductionthumbs and ratingsUser feedbacksignalsamplesAnnotationexpert labelscalibrateLLM judgescores at scaletrendsImprovementprompts, data, model

How it works

User-centered feedback

Feedback interfaces capture what real users think of real answers. Keep them light so users actually respond.

  • Thumbs up or down on each answer, with an optional reason list such as "wrong", "incomplete", "off topic".
  • Rating scales (for example 1 to 5) for helpfulness or clarity.
  • Implicit signals such as copying the answer, asking the same question again, or escalating to a human agent.

Store each rating with the request ID, prompt version, model ID and retrieved sources. Without that link, a low rating cannot be traced to its cause. A common pattern writes feedback to Amazon DynamoDB or Amazon S3 and analyzes it with Amazon Athena or dashboards.

Annotation workflows

Ratings tell you that an answer was bad; annotation tells you why. Reviewers label a sample of responses against a rubric: factually wrong, missing citation, unsafe, wrong tone.

ToolRole
Amazon SageMaker Ground TruthManaged labeling jobs with your own private workforce or vendors
Amazon Augmented AI (A2I)Human review loops triggered by a condition, such as low confidence
Amazon Bedrock human evaluation jobYour work team rates or compares responses from up to two models

Bedrock human evaluation jobs use a team you bring, with rating methods such as thumbs up or down, Likert scales, choice buttons and ordinal ranking. Prompt datasets are limited to 1,000 prompts per job.

LLM-as-a-Judge

A judge model reads the prompt, the response and optionally a reference answer, then returns a score and an explanation. Amazon Bedrock runs this as a model evaluation job with built-in metrics (correctness, completeness, faithfulness, helpfulness, logical coherence, relevance, following instructions, harmfulness, stereotyping, refusal and more) or custom metrics.

Calibrate the judge before you trust it. Have humans label a sample, run the judge on the same sample, and check that they agree. Use a judge model that is different from, and usually stronger than, the model being judged.

Exam tip

Judges can favor longer answers or their own style. A rubric with explicit criteria and a reference answer reduces this bias.

RAG evaluation

A RAG answer can fail in retrieval or in generation, so Bedrock evaluates both.

Job typeBuilt-in metricsNeeds ground truth?
Retrieve onlyContext relevance, context coverageCoverage needs ground truth texts
Retrieve and generateCorrectness, completeness, helpfulness, logical coherence, faithfulness, citation precision, citation coverage, harmfulness, stereotyping, refusalRecommended for correctness

You can evaluate an Amazon Bedrock Knowledge Base directly, or bring inference responses from an external RAG system. Low context relevance points at retrieval; high context relevance with low faithfulness points at the generator.

Agent performance

Agents plan, call tools and loop over many steps. Judge the path as well as the result.

What to measureExample signal
Task completion rateDid the session achieve the user's goal?
Tool usage effectivenessRight tool chosen, right parameters, no wasted calls
Reasoning qualityLogical steps, no contradictions, correct use of tool results
EfficiencyNumber of steps, tokens and latency per completed task

Amazon Bedrock AgentCore Evaluations scores agent traces with LLM-as-a-Judge. It works with agents on AgentCore Runtime and agents hosted elsewhere when they send OpenTelemetry traces, including Strands and LangGraph agents.

Built-in evaluators work at three levels:

  • Session level: goal success rate, which measures whether the whole conversation reached the goal.
  • Trace level: one turn, such as correctness, faithfulness, helpfulness, coherence, conciseness, instruction following and response relevance.
  • Tool level: tool selection accuracy and tool parameter accuracy for each tool call.
AgentStrands or LangGraphOpenTelemetry tracesAgentCore Evaluationsjudge modelscoresCloudWatchlogs and metrics

Online evaluation continuously scores live agent traffic and writes results to Amazon CloudWatch Logs and CloudWatch metrics. On-demand evaluation scores chosen sessions or traces, which suits testing before release. You can add custom evaluators for business rules.

For Amazon Bedrock Agents, turning on the trace in InvokeAgent shows the orchestration steps, the rationale and the action group calls, which is how you inspect reasoning in a multi step workflow.

Common mistakes

Collecting ratings without context. A thumbs down with no request ID, prompt version or sources cannot be acted on.

Trusting an uncalibrated judge. Check judge scores against human labels before using them as a release gate.

Judging an agent only by its final answer. An agent that reaches the right answer after ten redundant tool calls still has a cost and latency problem. Score tool selection and parameters too.

Using the same model as generator and judge. The judge tends to agree with itself. Use a different judge model.

What to remember for the exam

  • Combine user feedback, LLM judges and human annotation for a multi perspective view.
  • SageMaker Ground Truth labels data; Amazon A2I adds human review loops; Bedrock human evaluation jobs compare up to two models with your own team.
  • Bedrock RAG evaluation: retrieve only (context relevance, context coverage) or retrieve and generate (adds faithfulness, citation precision and coverage, correctness and more).
  • AgentCore Evaluations levels: session (goal success rate), trace (response quality) and tool (tool selection and parameter accuracy).
  • Online evaluation scores live traffic into CloudWatch; on-demand evaluation scores selected traces.

Check yourself

  • A RAG app shows high context relevance but low faithfulness. Where is the problem?

    Retrieval is finding the right content, so the generator is adding unsupported claims; fix the prompt or model, not the index.

  • Which AgentCore evaluator level checks whether an agent picked the correct tool?

    The tool level, through tool selection accuracy, with tool parameter accuracy checking the arguments.

  • How do you know an LLM judge is reliable enough to gate releases?

    Compare its scores with human labels on the same sample and confirm they agree before relying on it.

Topic 2 of 6 in Testing, Validation, and Troubleshooting

Test yourself on Testing, Validation, and Troubleshooting

The full AWS AIP-C01 question bank has 827 questions, with timed practice exams and an explanation for every answer.

This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.