Model Development · Topic 5 of 5
Classification and Regression Metrics
Databricks ML Associate: use F1, log loss, ROC AUC, RMSE, MAE and R squared, pick the metric for the goal, and exponentiate log scale predictions.
- 5 min
- F1 is the harmonic mean of precision and recall and suits imbalanced classes.
- Log loss punishes confident wrong probabilities; lower is better.
- ROC AUC is threshold independent; 0.5 is random, 1.0 is perfect.
What it is
A metric turns predictions into one number you can compare across models. Classification metrics judge predicted classes or probabilities; regression metrics judge the size of numeric errors.
The right metric follows the business goal, not habit. If the label was log-transformed for training, predictions must be converted back to the original scale before you compute errors or explain them.
How it works
Classification metrics
Start from the confusion matrix: true positives (TP), false positives (FP), false negatives (FN) and true negatives (TN).
| Metric | Meaning | Use when |
|---|---|---|
| Accuracy | Share of all predictions that are right | Classes are balanced and errors cost the same |
| Precision | TP / (TP + FP), how many flagged items are real | False alarms are expensive |
| Recall | TP / (TP + FN), how many real items are caught | Missing a positive is expensive |
| F1 | Harmonic mean of precision and recall | Imbalanced classes, both errors matter |
| Log loss | Penalizes confident wrong probabilities | Calibrated probabilities matter |
| ROC AUC | Ranking quality across all thresholds | Comparing models independent of a threshold |
Worked numbers: TP 40, FP 10, FN 20. Precision is 40 / 50 = 0.80 and recall is 40 / 60 = 0.67, so F1 = 2 x 0.80 x 0.67 / (0.80 + 0.67), about 0.73.
ROC AUC is 0.5 for random guessing and 1.0 for a perfect ranking. Log loss is lower when better and grows sharply when the model gives a high probability to the wrong class.
Regression metrics
| Metric | Meaning | Behavior |
|---|---|---|
| MAE | Average absolute error | Same units as the label, each error counts equally |
| RMSE | Square root of the average squared error | Same units, large errors weigh much more |
| MSE | Average squared error | Squared units, hard to explain |
| R squared | Share of the label's variance the model explains | 1 is perfect, 0 equals predicting the mean, can be negative |
Worked numbers: errors of 1, 1, 1 and 9. MAE is 12 / 4 = 3. RMSE is the square root of (1 + 1 + 1 + 81) / 4 = 21, about 4.58. The single large miss pulls RMSE well above MAE.
Metrics in Spark ML
| Evaluator | metricName options | Default |
|---|---|---|
| BinaryClassificationEvaluator | areaUnderROC, areaUnderPR | areaUnderROC |
| MulticlassClassificationEvaluator | f1, accuracy, weightedPrecision, weightedRecall, logLoss and others | f1 |
| RegressionEvaluator | rmse, mse, mae, r2, var | rmse |
from pyspark.ml.evaluation import RegressionEvaluator
ev = RegressionEvaluator(labelCol="price", predictionCol="prediction", metricName="r2")
r2 = ev.evaluate(preds_df)Choosing the metric for the scenario
| Scenario objective | Best fit |
|---|---|
| Fraud detection where missing fraud is costly | Recall, or F1 to balance it with precision |
| Spam filter where blocking a real email is costly | Precision |
| Rare positive class, one number to compare | F1 or area under the precision recall curve |
| Rank customers by churn risk, threshold not set yet | ROC AUC |
| Probabilities feed a pricing formula | Log loss |
| Forecast where big misses are very costly | RMSE |
| Forecast explained to business users in units | MAE |
| How much of the variation the model explains | R squared |
Exponentiating log-transformed labels
A skewed label such as price is often trained as log(price). The model then predicts log prices, so an RMSE computed on those values is in log units and cannot be compared with a model trained on raw prices.
from pyspark.sql.functions import col, exp
preds_df = model.transform(test_df).withColumn("prediction", exp(col("log_prediction")))Here the model's predictionCol was set to log_prediction. If the label was made with log1p, invert it with expm1 instead. A log prediction of 12.0 means about 162,755 dollars once exponentiated, which is the number to report.
How the exam asks about this
- "A cancer screening model must miss as few positive cases as possible." Optimize recall.
- "Compare two price models, one trained on log price." Exponentiate its predictions, then compute RMSE on both in dollars.
- "Errors of a few large outliers are what hurt the business." Use RMSE over MAE.
What to remember for the exam
- F1 is the harmonic mean of precision and recall and suits imbalanced classes.
- Log loss punishes confident wrong probabilities; lower is better.
- ROC AUC is threshold independent; 0.5 is random, 1.0 is perfect.
- RMSE weighs large errors more than MAE; both are in label units.
- R squared is the share of variance explained and can be negative for a poor model.
- Exponentiate log predictions before computing errors or explaining them.
Accuracy on an imbalanced dataset is misleading. A model that never predicts the rare class can score 99% accuracy with zero recall.
Common mistakes
Comparing RMSE across label scales. An RMSE of 0.3 on log price and 25,000 on raw price are not comparable. Bring both models back to the original units first.
Using ROC AUC when a fixed threshold drives decisions. AUC averages over every threshold. When the business acts at one cut off, check precision and recall at that cut off.
Treating R squared as accuracy. An R squared of 0.8 means 80% of the variance is explained, not that 80% of predictions are correct. It says nothing about the size of individual errors.
Check yourself
Precision is 0.5 and recall is 1.0. What is F1?
2 x 0.5 x 1.0 / 1.5, about 0.67, because the harmonic mean is pulled toward the lower value.
Why exponentiate predictions from a model trained on log price?
The model predicts log values, so metrics and explanations are only meaningful in dollars after reversing the log transform.
Which regression metric should you pick when large errors are especially costly?
RMSE, because squaring the errors makes large misses weigh much more than in MAE.
Topic 5 of 5 in Model Development
Test yourself on Model Development
The full Databricks Machine Learning Associate question bank has 600 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Databricks. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.