Applications of Foundation Models · Topic 7 of 7
Evaluating FM Performance
Evaluate foundation models with human review, benchmark datasets and metrics such as ROUGE, BLEU and BERTScore, and link results to business objectives.
- 4 min
- Human evaluation is the most reliable for open ended output, but slow and costly.
- A benchmark dataset gives a fair comparison between models on the same prompts.
- ROUGE is mainly for summarisation, and BLEU is mainly for translation.
What it is
Foundation model evaluation checks whether a model's output is good enough for your task. Unlike a classifier, a GenAI model gives free form text, so there is often no single correct answer to compare with.
That is why evaluation mixes several approaches: people judging output, automatic metrics, and tests against agreed benchmark sets.
How it works
Human evaluation
People read the answers and rate them for qualities such as accuracy, helpfulness, tone, relevance and safety. Human review is the most reliable way to judge open ended output, but it is slow and costly, and different reviewers can disagree. A clear scoring guide and several reviewers help.
Benchmark datasets
A benchmark dataset is a fixed collection of test prompts with known good answers or ratings. Running different models on the same benchmark lets you compare them fairly and measure progress. Public benchmarks cover areas such as general language understanding, question answering and safety, and you can also build your own from real company tasks.
A benchmark is only useful if it matches your use case. A model that scores high on a general test may still do poorly on your specialist content.
Automatic metrics
Automatic metrics compare model output with a reference text written by people.
| Metric | Typical use | Idea |
|---|---|---|
| ROUGE (Recall Oriented Understudy for Gisting Evaluation) | Summarisation | Measures how much of the reference text's wording appears in the output |
| BLEU (Bilingual Evaluation Understudy) | Machine translation | Measures how closely the output's word sequences match the reference translation |
| BERTScore | Many text tasks | Compares meaning using embeddings, so it can credit wording that differs but means the same |
ROUGE and BLEU count overlap of words, so they can miss a good answer phrased differently. BERTScore looks at semantic similarity instead, which makes it better for paraphrases. Perplexity is another measure, which describes how well a language model predicts text.
Evaluation on Amazon Bedrock
Amazon Bedrock includes model evaluation features. You can run automatic evaluation with built in metrics and datasets for tasks such as summarisation, question answering and classification.
You can also set up human evaluation, using your own team or an AWS managed team, to rate answers against criteria you choose. Amazon SageMaker Clarify also offers evaluation of foundation models.
Evaluating RAG applications
For RAG, check both parts. Retrieval quality asks whether the right passages were found, and generation quality asks whether the answer stays faithful to those passages.
Judging business effectiveness
A model with good scores can still fail the business. Link evaluation to the goals set at the start.
| Goal | Business measure |
|---|---|
| Productivity | Time saved per task |
| Customer experience | Satisfaction scores, resolution rate |
| Revenue | Conversion or sales uplift |
| Cost | Cost per answer, fewer escalations |
| Quality | Error rate found in review |
Run a pilot with real users and compare these measures with the situation before GenAI.
Safety and fairness checks
Evaluation should include tests for harmful content, bias and leakage of private data. Test with tricky prompts, including attempts to bypass the rules, and review the results.
Keep evaluating
Evaluate again after any change to the model, prompt or data, and keep monitoring in production. Quality can shift as users and content change.
What to remember for the exam
- Human evaluation is the most reliable for open ended output, but slow and costly.
- A benchmark dataset gives a fair comparison between models on the same prompts.
- ROUGE is mainly for summarisation, and BLEU is mainly for translation.
- BERTScore measures semantic similarity using embeddings.
- ROUGE and BLEU rely on word overlap, so they can miss valid rewording.
- Amazon Bedrock model evaluation supports automatic and human evaluation.
- RAG needs checks on both retrieval and the final answer.
- Always connect evaluation results to business objectives.
Summarisation scored by recall of reference words is ROUGE, translation scored by matching word sequences is BLEU, and a meaning based comparison is BERTScore.
Common mistakes
Swapping ROUGE and BLEU. ROUGE is the usual summarisation metric, and BLEU is the usual translation metric.
Trusting a public benchmark without checking relevance. Test on tasks like your own.
Relying only on automatic metrics. They can miss tone, safety and factual problems.
Ignoring business results. A good score does not prove the model meets the goal.
Evaluating once. Results change when models, prompts or data change.
Check yourself
Which metric is commonly used to evaluate summarisation?
ROUGE.
Why is BERTScore useful?
It compares meaning using embeddings, so it credits answers that are worded differently but say the same thing.
Why use human evaluation?
People can judge qualities such as helpfulness, tone and accuracy that automatic metrics miss.
Topic 7 of 7 in Applications of Foundation Models
Test yourself on Applications of Foundation Models
The full AWS AIF-C01 question bank has 849 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.