Testing, Validation, and Troubleshooting · Topic 1 of 6
Assessment Metrics and Model Evaluation
Measure FM output quality with relevance, accuracy, consistency and fluency, then pick models with Bedrock evaluations, A/B and canary tests and cost data.
- 6 min
- Includes a diagram
- FM quality is multi dimensional: relevance, factual accuracy, consistency, fluency and safety.
- Bedrock evaluation job types: automatic (accuracy, robustness, toxicity), model as a judge, human based, and RAG evaluation.
- Datasets go in S3 as JSON Lines; reports go to an S3 bucket you name.
What it is
Traditional ML evaluation compares a prediction with one correct label, using accuracy, precision or F1. A foundation model returns free text, so there is rarely a single right string. Two answers can use different words and both be correct.
An FM assessment framework therefore scores qualities of a response: is it relevant, factually accurate, consistent and fluent. A model evaluation system then uses those scores, plus cost and latency, to choose the best model, prompt and inference settings for one workload.
How it works
Quality dimensions beyond classic ML metrics
| Dimension | Question it answers | Typical way to measure |
|---|---|---|
| Relevance | Does the answer address what was asked? | LLM judge score, human rating |
| Factual accuracy | Are the claims true and supported? | Compare with a reference answer, or check faithfulness to supplied context |
| Consistency | Does the same input give equivalent answers across runs and versions? | Run each prompt several times and compare meaning, not exact text |
| Fluency | Is the text readable, grammatical and well formed? | LLM judge or human rating |
| Safety | Is the answer free of harmful or biased content? | Toxicity, harmfulness, stereotyping metrics |
Exact string match fails for generative output. Use semantic comparison instead: embedding similarity, a reference based metric such as BERTScore, or a judge model that reads both answers.
When a scenario says "beyond traditional ML evaluation", the answer usually involves an LLM judge, human review or semantic similarity. Accuracy against a fixed label alone is the trap.
Amazon Bedrock evaluations
Amazon Bedrock offers managed evaluation jobs. You supply a prompt dataset in Amazon S3 (JSON Lines), Bedrock runs the models, and the report lands in an S3 bucket you choose.
| Job type | How scoring works | Use it when |
|---|---|---|
| Automatic (programmatic) | Computed metrics: accuracy, robustness, toxicity | Quick, cheap baseline on a standard task type |
| Model as a judge | A second LLM scores each response and explains why | You need nuanced quality scores at scale |
| Human based | Your own work team rates responses | Subjective quality, brand voice, domain expertise |
| RAG evaluation | Judge model scores retrieval and generated answers | Comparing knowledge bases or RAG setups |
Automatic jobs offer four task types: general text generation, text summarization, question and answer, and text classification. Built-in datasets exist for each task, or you bring your own prompts.
Judge model jobs offer built-in metrics such as correctness, completeness, faithfulness, helpfulness, logical coherence, relevance, following instructions, professional style and tone, harmfulness, stereotyping and refusal. You can also define custom metrics with your own judge prompt and rating scale.
A key option is bring your own inference responses. You put responses that another model already produced into the dataset, and Bedrock skips the model call and only scores them. This is how you compare a Bedrock model with a model hosted elsewhere in one report.
Multi model and configuration comparison
Run the same dataset against several candidates and compare them on the same metrics. A candidate is not only a model; it is a model plus prompt version plus inference parameters such as temperature and maximum tokens.
- Offline evaluation uses a fixed dataset before release, so results are repeatable.
- A/B testing splits live users between two configurations and compares quality signals and business outcomes.
- Canary testing sends a small share of live traffic to the new configuration, watches error rates, latency and quality, then increases the share or rolls back.
Bedrock does not split traffic between models for you. The application or a routing layer does it, for example with weighted Lambda aliases, an API Gateway canary release or an AWS AppConfig feature flag. Amazon SageMaker AI endpoints support production variants with traffic weights and shadow tests for self hosted models.
Cost and performance analysis
The best configuration is rarely the highest scoring one. The exam expects you to balance quality with cost and speed.
| Measure | Where it comes from | What it tells you |
|---|---|---|
| Token efficiency | InputTokenCount and OutputTokenCount metrics, usage in the Converse response | Cost per request and per task |
| Latency | InvocationLatency, TimeToFirstToken for streaming | User experience |
| Latency to quality ratio | Evaluation score compared with latency | Whether a slower model earns its delay |
| Business outcome | Conversion, deflection, resolution rate | Whether the change matters to the business |
When a scenario stresses cost, a smaller model that still clears the quality bar is usually the answer. Pay for a larger model only when a named quality requirement demands it.
Common mistakes
Grading generative output with exact match. Two correct answers rarely share wording. Exact match scores them as failures and pushes you toward the wrong model.
Picking the model with the top quality score. If the requirement says MOST cost-effective while meeting a quality bar, choose the cheapest configuration that clears the bar.
Expecting Bedrock to run the canary. Traffic splitting between Bedrock models is application logic. Bedrock evaluation jobs are offline, dataset based comparisons.
Evaluating with a toy dataset. A few hand-picked prompts hide failure modes. Use real user questions and edge cases.
What to remember for the exam
- FM quality is multi dimensional: relevance, factual accuracy, consistency, fluency and safety.
- Bedrock evaluation job types: automatic (accuracy, robustness, toxicity), model as a judge, human based, and RAG evaluation.
- Datasets go in S3 as JSON Lines; reports go to an S3 bucket you name.
- Bring your own inference responses lets you score models hosted outside Bedrock.
- A/B and canary tests run on live traffic and are built by your routing layer; evaluation jobs are offline.
- Compare candidates on quality, tokens, latency and business outcome together.
Robustness in an automatic job means how much output changes when the input is slightly perturbed. It is not the same as consistency across repeated runs of an identical prompt.
Check yourself
Why is exact string match a poor metric for FM output?
Correct answers can be worded in many ways, so you need semantic comparison, a judge model or human rating instead.
How can one Bedrock evaluation report compare a Bedrock model with a model hosted on another platform?
Supply the external model's answers as your own inference responses in the dataset, and Bedrock scores them without invoking a model.
A team wants to try a new model on 5 percent of production traffic and roll back on quality loss. What is this and who routes the traffic?
It is a canary test, and the application or routing layer, such as weighted Lambda aliases or a feature flag, splits the traffic.
Topic 1 of 6 in Testing, Validation, and Troubleshooting
Test yourself on Testing, Validation, and Troubleshooting
The full AWS AIP-C01 question bank has 827 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.