Testing, Validation, and Troubleshooting · Topic 2 of 6
Human Feedback, LLM-as-a-Judge and Agent Evaluation
Collect user ratings and annotations, score outputs with LLM-as-a-Judge and RAG evaluation, and measure agent goal success and tool use with AgentCore.
- 5 min
- Combine user feedback, LLM judges and human annotation for a multi perspective view.
- SageMaker Ground Truth labels data; Amazon A2I adds human review loops; Bedrock human evaluation jobs compare up to two models with your own team.
- Bedrock RAG evaluation: retrieve only (context relevance, context coverage) or retrieve and generate (adds faithfulness, citation precision and coverage, correctness and more).
What it is
No single evaluator sees everything. Automated metrics are fast but shallow, judge models scale but can be biased, and humans are accurate but slow and costly.
A thorough assessment combines three perspectives: users who rate real answers, LLM judges that score every response against a rubric, and expert reviewers who annotate samples. Agents add a fourth layer, because you must judge the steps and tool calls, not only the final answer.
How it works
User-centered feedback
Feedback interfaces capture what real users think of real answers. Keep them light so users actually respond.
- Thumbs up or down on each answer, with an optional reason list such as "wrong", "incomplete", "off topic".
- Rating scales (for example 1 to 5) for helpfulness or clarity.
- Implicit signals such as copying the answer, asking the same question again, or escalating to a human agent.
Store each rating with the request ID, prompt version, model ID and retrieved sources. Without that link, a low rating cannot be traced to its cause. A common pattern writes feedback to Amazon DynamoDB or Amazon S3 and analyzes it with Amazon Athena or dashboards.
Annotation workflows
Ratings tell you that an answer was bad; annotation tells you why. Reviewers label a sample of responses against a rubric: factually wrong, missing citation, unsafe, wrong tone.
| Tool | Role |
|---|---|
| Amazon SageMaker Ground Truth | Managed labeling jobs with your own private workforce or vendors |
| Amazon Augmented AI (A2I) | Human review loops triggered by a condition, such as low confidence |
| Amazon Bedrock human evaluation job | Your work team rates or compares responses from up to two models |
Bedrock human evaluation jobs use a team you bring, with rating methods such as thumbs up or down, Likert scales, choice buttons and ordinal ranking. Prompt datasets are limited to 1,000 prompts per job.
LLM-as-a-Judge
A judge model reads the prompt, the response and optionally a reference answer, then returns a score and an explanation. Amazon Bedrock runs this as a model evaluation job with built-in metrics (correctness, completeness, faithfulness, helpfulness, logical coherence, relevance, following instructions, harmfulness, stereotyping, refusal and more) or custom metrics.
Calibrate the judge before you trust it. Have humans label a sample, run the judge on the same sample, and check that they agree. Use a judge model that is different from, and usually stronger than, the model being judged.
Judges can favor longer answers or their own style. A rubric with explicit criteria and a reference answer reduces this bias.
RAG evaluation
A RAG answer can fail in retrieval or in generation, so Bedrock evaluates both.
| Job type | Built-in metrics | Needs ground truth? |
|---|---|---|
| Retrieve only | Context relevance, context coverage | Coverage needs ground truth texts |
| Retrieve and generate | Correctness, completeness, helpfulness, logical coherence, faithfulness, citation precision, citation coverage, harmfulness, stereotyping, refusal | Recommended for correctness |
You can evaluate an Amazon Bedrock Knowledge Base directly, or bring inference responses from an external RAG system. Low context relevance points at retrieval; high context relevance with low faithfulness points at the generator.
Agent performance
Agents plan, call tools and loop over many steps. Judge the path as well as the result.
| What to measure | Example signal |
|---|---|
| Task completion rate | Did the session achieve the user's goal? |
| Tool usage effectiveness | Right tool chosen, right parameters, no wasted calls |
| Reasoning quality | Logical steps, no contradictions, correct use of tool results |
| Efficiency | Number of steps, tokens and latency per completed task |
Amazon Bedrock AgentCore Evaluations scores agent traces with LLM-as-a-Judge. It works with agents on AgentCore Runtime and agents hosted elsewhere when they send OpenTelemetry traces, including Strands and LangGraph agents.
Built-in evaluators work at three levels:
- Session level: goal success rate, which measures whether the whole conversation reached the goal.
- Trace level: one turn, such as correctness, faithfulness, helpfulness, coherence, conciseness, instruction following and response relevance.
- Tool level: tool selection accuracy and tool parameter accuracy for each tool call.
Online evaluation continuously scores live agent traffic and writes results to Amazon CloudWatch Logs and CloudWatch metrics. On-demand evaluation scores chosen sessions or traces, which suits testing before release. You can add custom evaluators for business rules.
For Amazon Bedrock Agents, turning on the trace in InvokeAgent shows the orchestration steps, the rationale and the action group calls, which is how you inspect reasoning in a multi step workflow.
Common mistakes
Collecting ratings without context. A thumbs down with no request ID, prompt version or sources cannot be acted on.
Trusting an uncalibrated judge. Check judge scores against human labels before using them as a release gate.
Judging an agent only by its final answer. An agent that reaches the right answer after ten redundant tool calls still has a cost and latency problem. Score tool selection and parameters too.
Using the same model as generator and judge. The judge tends to agree with itself. Use a different judge model.
What to remember for the exam
- Combine user feedback, LLM judges and human annotation for a multi perspective view.
- SageMaker Ground Truth labels data; Amazon A2I adds human review loops; Bedrock human evaluation jobs compare up to two models with your own team.
- Bedrock RAG evaluation: retrieve only (context relevance, context coverage) or retrieve and generate (adds faithfulness, citation precision and coverage, correctness and more).
- AgentCore Evaluations levels: session (goal success rate), trace (response quality) and tool (tool selection and parameter accuracy).
- Online evaluation scores live traffic into CloudWatch; on-demand evaluation scores selected traces.
Check yourself
A RAG app shows high context relevance but low faithfulness. Where is the problem?
Retrieval is finding the right content, so the generator is adding unsupported claims; fix the prompt or model, not the index.
Which AgentCore evaluator level checks whether an agent picked the correct tool?
The tool level, through tool selection accuracy, with tool parameter accuracy checking the arguments.
How do you know an LLM judge is reliable enough to gate releases?
Compare its scores with human labels on the same sample and confirm they agree before relying on it.
Topic 2 of 6 in Testing, Validation, and Troubleshooting
Test yourself on Testing, Validation, and Troubleshooting
The full AWS AIP-C01 question bank has 827 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.