Fundamentals of AI and ML · Topic 7 of 8
Model Performance Metrics
Read the confusion matrix and choose between accuracy, precision, recall, F1 score, AUC and error metrics, and link model scores to business metrics.
- 4 min
- A confusion matrix has TP, FP, FN and TN.
- Accuracy is the share of all correct predictions, and it can mislead on unbalanced data.
- Precision is TP divided by TP plus FP, and it matters when false alarms are costly.
What it is
Evaluation tells you whether a trained model is good enough to use. Different tasks need different metrics, and the right metric depends on which kind of mistake costs the most.
There are two layers of measurement. Technical metrics describe how well the model predicts, and business metrics describe whether it helps the organisation.
How it works
The confusion matrix
For a classification task with a positive class, such as fraud, every prediction falls in one of four boxes.
| Predicted positive | Predicted negative | |
|---|---|---|
| Actually positive | True positive (TP) | False negative (FN) |
| Actually negative | False positive (FP) | True negative (TN) |
A false positive is a false alarm. A false negative is a miss, where the model fails to catch a real positive.
Classification metrics
| Metric | Question it answers | Formula |
|---|---|---|
| Accuracy | How often is the model right overall | (TP + TN) / all predictions |
| Precision | Of the items flagged positive, how many truly are | TP / (TP + FP) |
| Recall | Of all real positives, how many were found | TP / (TP + FN) |
| F1 score | A single balance of precision and recall | Harmonic mean of the two |
| AUC | How well the model separates the classes across thresholds | Area under the ROC curve |
AUC stands for area under the curve, and the curve is the receiver operating characteristic (ROC) curve. A value near 1 means very good separation, and a value near 0.5 means no better than guessing.
Choosing the right metric
Accuracy can mislead when classes are unbalanced. If only 1 transaction in 1,000 is fraud, a model that always says not fraud is 99.9 percent accurate and catches nothing.
- Choose recall when missing a real case is costly, such as a disease screening or fraud check.
- Choose precision when false alarms are costly, such as blocking a good customer by mistake.
- Choose F1 when both matter and you want one number.
Regression metrics
Regression predicts numbers, so metrics measure the size of the error.
| Metric | Meaning |
|---|---|
| Mean absolute error (MAE) | Average size of the errors |
| Mean squared error (MSE) | Average of squared errors, which punishes big misses |
| Root mean squared error (RMSE) | Square root of MSE, in the same unit as the target |
| R squared | Share of variation in the target that the model explains |
Lower error is better. A higher R squared is better.
Business metrics
A good technical score does not guarantee value. Link the model to the business goal set at the start.
| Business metric | Example use |
|---|---|
| Cost savings | Fewer manual reviews |
| Revenue or conversion | More purchases from recommendations |
| Customer satisfaction | Faster answers from a chatbot |
| Efficiency | Less time to process a document |
| Risk reduction | Fewer undetected frauds |
Evaluation on AWS
Amazon SageMaker AI reports training and validation metrics for the models you train. Amazon SageMaker Clarify can check a model for bias and explain predictions, and it comes up again in the responsible AI chapter.
What to remember for the exam
- A confusion matrix has TP, FP, FN and TN.
- Accuracy is the share of all correct predictions, and it can mislead on unbalanced data.
- Precision is TP divided by TP plus FP, and it matters when false alarms are costly.
- Recall is TP divided by TP plus FN, and it matters when misses are costly.
- F1 balances precision and recall.
- AUC close to 1 means the model separates classes well, and 0.5 is no better than chance.
- MAE, MSE and RMSE measure error for regression, and lower is better.
- Always tie technical results to a business metric.
Ask which error hurts more. If missing a case is worse, pick recall, and if a false alarm is worse, pick precision.
Common mistakes
Trusting accuracy on rare events. When positives are rare, a useless model can still look highly accurate.
Swapping precision and recall. Precision looks at the flagged items. Recall looks at the real positives.
Using classification metrics for regression. Accuracy and F1 do not suit a numeric target. Use an error metric.
Judging success on the model score alone. A model can score well and still miss the business goal.
Reading a lower error as worse. For error metrics, lower values are better.
Check yourself
A cancer screening model must catch as many real cases as possible. Which metric matters most?
Recall, since a missed case is the costly mistake.
Why can accuracy be misleading?
With rare positive cases, a model that never predicts them can still be highly accurate while catching nothing.
What does RMSE measure?
The typical size of a regression model's prediction errors, in the same unit as the target.
Topic 7 of 8 in Fundamentals of AI and ML
Test yourself on Fundamentals of AI and ML
The full AWS AIF-C01 question bank has 849 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.