Model Development · Topic 4 of 5
Cross Validation and Grid Search
Databricks ML Associate: cross validation vs a train validation split, CrossValidator in Spark ML, and counting the models a grid search trains.
- 5 min
- Includes a diagram
- Cross-validation averages k scores per setting; a train-validation split uses one score.
- Cross-validation is more reliable but costs about k times more compute.
- CrossValidator needs an estimator, a param grid and an evaluator; numFolds defaults to 3.
What it is
To pick hyperparameters you need a score on data the model did not train on. A train-validation split holds out one part of the training data once. Cross-validation (k-fold) splits the training data into k folds, trains k times, and each time scores on the fold that was left out.
Spark ML offers both as estimators: TrainValidationSplit and CrossValidator. Each takes a model or pipeline, a grid of hyperparameter settings and an evaluator, then refits the best setting on all the training data.
How it works
Cross-validation versus a train-validation split
| Train-validation split | k-fold cross-validation | |
|---|---|---|
| Models per setting | 1 | k |
| Score per setting | One validation score | Average of k scores |
| Reliability | Depends on one random split | More stable, every row is validated once |
| Compute cost | Low | About k times higher |
| Best for | Large data, tight time budget | Smaller data, when a fair comparison matters |
The benefit of cross-validation is a more reliable estimate that uses all the training rows for both training and validation. The downside is cost: with 5 folds, every setting trains 5 models instead of 1.
Keep a separate test set outside both methods. Cross-validation chooses hyperparameters; the test set gives the final, unbiased score.
Performing cross-validation in Spark ML
from pyspark.ml.tuning import CrossValidator, ParamGridBuilder
from pyspark.ml.evaluation import RegressionEvaluator
grid = (ParamGridBuilder().addGrid(rf.maxDepth, [4, 8, 12])
.addGrid(rf.numTrees, [50, 100]).build())
cv = CrossValidator(estimator=pipeline, estimatorParamMaps=grid,
evaluator=RegressionEvaluator(labelCol="price", metricName="rmse"),
numFolds=3, parallelism=4, seed=42)
cv_model = cv.fit(train_df)cv.fit() returns a CrossValidatorModel. Its bestModel is the pipeline refitted on all of train_df with the best setting, and avgMetrics holds the average score of each grid setting in grid order.
| Parameter | Meaning |
|---|---|
estimator | The model or whole pipeline to tune |
estimatorParamMaps | The grid from ParamGridBuilder |
evaluator | The metric used to compare settings |
numFolds | k, default 3 |
parallelism | How many models train at the same time, default 1 |
The evaluator knows whether larger is better: RMSE is minimized, R squared and AUC are maximized. TrainValidationSplit has the same shape with trainRatio (default 0.75) instead of numFolds.
Pipeline inside cross-validation, or cross-validation inside the pipeline
| Placement | What happens | Trade-off |
|---|---|---|
| Pipeline as the CrossValidator estimator | Every fold refits all preprocessing stages | No leakage between folds, more work |
| CrossValidator as the last pipeline stage | Preprocessing is fitted once on all training data | Faster, but fold scores see preprocessing learned from held out rows |
Put the whole pipeline inside the CrossValidator when preprocessing learns statistics (imputer means, scaler values). Put the CrossValidator inside the pipeline when the earlier stages are cheap to share and you want speed.
Counting the models trained
The formula is grid settings x folds + 1 for cross-validation, and grid settings + 1 for a train-validation split. The +1 is the final refit of the best setting on all training data.
| Grid | Method | Models trained |
|---|---|---|
| 3 maxDepth x 2 numTrees = 6 settings | 3-fold CrossValidator | 6 x 3 + 1 = 19 |
| 6 settings | TrainValidationSplit | 6 + 1 = 7 |
| 4 x 3 x 2 = 24 settings | 5-fold CrossValidator | 24 x 5 + 1 = 121 |
| Hyperopt with max_evals=30, each trial using 3-fold CV | fmin | 30 x 3 = 90, plus any final refit you run |
Multiply the number of values of every parameter to get the settings first. Parallelism changes how fast the models train, not how many.
How the exam asks about this
- "A grid of 4 regParam values and 3 elasticNetParam values with 5 folds." That is 4 x 3 x 5 + 1 = 61 models.
- "The dataset is huge and tuning must finish quickly." Use a train-validation split, which trains one model per setting.
- "The tuned pipeline must score new data." Use
cv_model.bestModel, already refitted on all training data.
What to remember for the exam
- Cross-validation averages k scores per setting; a train-validation split uses one score.
- Cross-validation is more reliable but costs about k times more compute.
- CrossValidator needs an estimator, a param grid and an evaluator; numFolds defaults to 3.
- Models trained = settings x k + 1 for CrossValidator, settings + 1 for TrainValidationSplit.
- bestModel is refitted on the full training data; avgMetrics lists each setting's average score.
- Keep a separate test set for the final score.
Grid size is the product of the value counts, not their sum. Three parameters with 3, 4 and 5 values give 60 settings, not 12.
Common mistakes
Forgetting the final refit. After choosing the best setting, CrossValidator and TrainValidationSplit train one more model on all training data. Leave out the +1 and the count is wrong.
Reporting the cross-validation score as the final result. The best average was selected from many settings, so it is optimistic. Score bestModel on a held out test set.
Assuming parallelism reduces the work. Raising parallelism trains several models at once. The total number of models stays the same.
Check yourself
How many models does a 5-fold CrossValidator train over a grid of 2 x 5 settings?
10 settings x 5 folds = 50, plus one refit of the best setting, so 51 models.
What is the main downside of cross-validation compared with a train-validation split?
Compute cost, since each setting trains k models instead of one.
What does CrossValidatorModel.bestModel contain?
The estimator or pipeline refitted on all the training data with the best hyperparameter setting.
Topic 4 of 5 in Model Development
Test yourself on Model Development
The full Databricks Machine Learning Associate question bank has 600 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Databricks. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.