Model Development · Topic 3 of 5

Hyperparameter Tuning with Hyperopt

Databricks ML Associate: tune with Hyperopt fmin, compare random, grid and Bayesian search, and parallelize single node models with SparkTrials.

5 min

In a nutshell

fmin(fn, space, algo, max_evals, trials) minimizes the loss returned by fn.
Return a negative score when the metric should be maximized.
tpe.suggest is Bayesian search; rand.suggest is random search.

What it is

Hyperparameters are settings you choose before training, such as tree depth, number of trees or regularization strength. Tuning tries many settings and keeps the one with the best validation score.

Hyperopt is a Python library whose fmin function searches a space of hyperparameters to minimize a loss. On Databricks, SparkTrials spreads the trials across the cluster so several single node models train at once.

Current status: the open source Hyperopt is no longer maintained and is not included in Databricks Runtime ML after 16.4 LTS ML. Databricks recommends Optuna for single node optimization or Ray Tune for distributed tuning in new work, but the exam objectives still use Hyperopt.

How it works

The arguments of fmin

Argument	What it is
`fn`	The objective function: takes one set of hyperparameters, trains, returns a loss
`space`	The search space built with `hp` expressions
`algo`	The search algorithm: `tpe.suggest` (Bayesian) or `rand.suggest` (random)
`max_evals`	How many settings to try in total
`trials` (optional)	`Trials()` to record results on the driver, or `SparkTrials()` to run in parallel

Hyperopt always minimizes. To maximize accuracy or AUC, return the negative value as the loss.

from hyperopt import fmin, tpe, hp, SparkTrials, STATUS_OK
space = {"max_depth": hp.quniform("max_depth", 2, 12, 1),
         "n_estimators": hp.choice("n_estimators", [100, 200, 400])}
def objective(params):
    acc = train_and_score(params)  # your single node training code
    return {"loss": -acc, "status": STATUS_OK}
best = fmin(fn=objective, space=space, algo=tpe.suggest,
            max_evals=48, trials=SparkTrials(parallelism=4))

The objective can return a plain number or a dictionary with loss and status. fmin returns the best values found; for an hp.choice parameter it returns the index of the option, and hyperopt.space_eval(space, best) turns that back into the real value.

Search space expressions

Expression	Samples
`hp.choice(label, options)`	One item from a list
`hp.uniform(label, low, high)`	A float spread evenly between low and high
`hp.quniform(label, low, high, q)`	Evenly spread values rounded to steps of q (returned as floats, so cast to int)
`hp.loguniform(label, low, high)`	A value whose log is uniform, good for learning rates and regularization

Random, grid and Bayesian search

Method	How it picks settings	Strength	Weakness
Grid search	Every combination of listed values	Exhaustive and repeatable	Model count multiplies with every parameter
Random search	Random draws from the space	Covers wide ranges cheaply	Ignores what earlier trials learned
Bayesian search (TPE)	Uses past results to choose promising settings	Fewer trials to reach a good result	Each choice depends on earlier results

Worked numbers: a grid of 4 depths x 3 tree counts x 3 learning rates is 36 settings, every one trained. With Hyperopt you could set max_evals=20 and let TPE focus on the best region. Grid search on Spark ML usually uses ParamGridBuilder; Hyperopt itself offers random (rand.suggest) and Bayesian (tpe.suggest) search.

Parallelizing single node models with SparkTrials

A single node library such as scikit-learn trains one model on one machine. SparkTrials runs each trial as a Spark task on a worker, so with parallelism=4 four models train at the same time.

Situation	Trials class
Single node model (scikit-learn, XGBoost on one node)	`SparkTrials` to run trials in parallel
Distributed Spark ML model	`Trials` (the default), since each training job already uses the cluster

Parallelism is a trade-off. With TPE, trials that run at the same time cannot learn from each other, so very high parallelism makes the search behave more like random search. More parallelism finishes sooner; fewer parallel trials let each one benefit from earlier results.

How the exam asks about this

"The objective returns accuracy and fmin finds the worst model." Hyperopt minimizes; return negative accuracy as the loss.
"Tune a scikit-learn model faster on a multi node cluster." Pass SparkTrials to fmin.
"Tune a Spark ML random forest with Hyperopt." Use the default Trials, not SparkTrials, because Spark ML already distributes training.

What to remember for the exam

fmin(fn, space, algo, max_evals, trials) minimizes the loss returned by fn.
Return a negative score when the metric should be maximized.
tpe.suggest is Bayesian search; rand.suggest is random search.
Bayesian search uses earlier results; random and grid search do not.
SparkTrials parallelizes single node models across workers; do not use it with distributed Spark ML training.
Hyperopt is no longer maintained; Databricks recommends Optuna or Ray Tune for new work.

Watch out

For an hp.choice parameter, the best dictionary from fmin holds the option's index, not its value. Use space_eval(space, best) before retraining the final model.

Common mistakes

Maximizing with fmin. fmin only minimizes. Returning accuracy as the loss makes it hunt for the least accurate model.

Using SparkTrials with Spark ML. Each Spark ML fit already runs across the cluster. SparkTrials is meant for single node libraries; with distributed training, use the default Trials.

Thinking grid search is always thorough enough. A grid only tests the values you list and grows multiplicatively. Random or Bayesian search often finds a better setting in fewer trials.

Check yourself

What does fmin do with the value the objective returns?
It treats it as a loss to minimize, so a metric to maximize must be returned as its negative.
Which algo argument gives Bayesian search in Hyperopt?
tpe.suggest, the Tree of Parzen Estimators, which uses earlier trial results to choose the next setting.
When should you pass SparkTrials to fmin?
When tuning a single node model such as scikit-learn, so trials run in parallel on the workers.

Topic 3 of 5 in Model Development

Test yourself on Model Development

The full Databricks Machine Learning Associate question bank has 600 questions, with timed practice exams and an explanation for every answer.

Get Full Access Try Sample Exam

This study guide is independent and is not affiliated with, endorsed by, or sponsored by Databricks. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.