Hyperopt Example

guildai · July 10, 2020, 6:34pm

Overview

This example illustrate how to create a custom optimizer using Hyperopt. Follow the patterns outlined below to use other sequential tuning algorithms with your project.

Project files:


guild.yml	Project Guild file
train.py	Sample training script
tpe.py	Optimizer support using Tree of Parzen Estimators with Hyperopt
requirements.txt	List of required libraries

An optimizer is a Guild operation that specializes in running multiple trials based on a batch prototype.

In this example, we create a custom optimizer named tpe. The optimizer is used to find optimal hyperparameter values using the Tree of Parzen Estimators algorithm from the Hyperopt library.

Here’s the Guild file that defines both the training operation and the optimizer:

github.com

guildai/guildai/blob/main/examples/hyperopt/guild.yml

train:
  flags-import: all
  optimizers:
    tpe:
      default: yes

tpe:
  flags-import:
    - prior_weight
    - EI_candidates
    - gamma
    - startup_jobs
  default-max-trials: 10

Note Each operation uses the Python module corresponding to its name. If a module uses a different name, specify it using the main attribute.

Start an optimization batch by running:

guild run train x=[-2:2] --optimize

The train operation is configured to use the tpe optimizer by default (see above).

The tpe optimizer is configured to run 10 trials by default. Change this value using --max-trials.

You can view operation help for tpe to show supported flags. These are set using -Fo options with guild run.

guild run tpe --help-op

Implementation

The tpe operation is implemented in tpe.py.

Implementation conists of the following steps:

Read settings for the batch run and the batch prototype.
Convert prototype flag values into a Hyperopt search space
Define a function to minimize that runs a batch trial using hyperparameter values provided by Hyperopt.
Use Hyperopt fmin to start a sequential optimization process using the search space and function to minimize.

Read Batch Settings

Use batch_util.batch_run() to get the current batch run. This is used to read the batch settings.

github.com

guildai/guildai/blob/main/examples/hyperopt/tpe.py#L24


      
          
          prior_weight = 1.0
          EI_candidates = 24
          gamma = 0.25
          startup_jobs = 20
          
          
          def main():
              batch_util.init_logging()
              batch = batch_util.batch_run()
              batch_flags = batch.get("flags")
              max_trials = batch.get("max_trials") or DEFAULT_MAX_TRIALS
              random_seed = batch.get("random_seed")
              trials = hyperopt.Trials()
              try:
                  hyperopt.fmin(
                      fn=_objective_fn(batch),
                      space=_space_for_flags(batch),
                      algo=_tpe_suggest,
                      max_evals=max_trials,
                      show_progressbar=False,

Convert Prototype Flags to Search Space

The batch proto run defines the flag values used for trials. These may contain search specs like [-2:2] or uniform[-2:2]. This syntax represents a Guild search space function. Decode search space functions to create a Hyperopt search space.

Read the proto flags and uses them to create a search space:

github.com

guildai/guildai/blob/main/examples/hyperopt/tpe.py#L69


      
                  return {
                      "status": hyperopt.STATUS_OK,
                      "loss": obj_value * obj_negate,
                      "run": run,
                  }
          
              return fn
          
          
          def _space_for_flags(batch):
              flags = batch.batch_proto.get("flags")
              return {name: _space_for_flag_val(val, name) for name, val in flags.items()}
          
          
          def _space_for_flag_val(val, label):
              if isinstance(val, list):
                  return hyperopt.hp.choice(label, val)
              else:
                  try:
                      func_name, func_args = flag_util.decode_flag_function(val)
                  except ValueError:

Decode each flag function to create a corresponding Hyperopt search space expression:

github.com

guildai/guildai/blob/main/examples/hyperopt/tpe.py#L79


      
              flags = batch.batch_proto.get("flags")
              return {name: _space_for_flag_val(val, name) for name, val in flags.items()}
          
          
          def _space_for_flag_val(val, label):
              if isinstance(val, list):
                  return hyperopt.hp.choice(label, val)
              else:
                  try:
                      func_name, func_args = flag_util.decode_flag_function(val)
                  except ValueError:
                      return hyperopt.hp.choice(label, [val])
                  else:
                      return _space_for_func(func_name, func_args, label, val)
          
          
          def _space_for_func(func_name, func_args, flag_name, flag_val):
              func_name = func_name or "uniform"
              try:
                  space_init = getattr(hyperopt.hp, func_name)
              except AttributeError:

Function to Minimize

Hyperopt calls a function with a dictionary of suggested flag values. Use batch_util.run_trial to generate a trial for the batch using the suggested values:

github.com

guildai/guildai/blob/main/examples/hyperopt/tpe.py#L48


      
              except hyperopt.exceptions.AllTrialsFailed:
                  pass
              else:
                  _label_best(trials.best_trial)
          
          
          def _objective_fn(batch):
              obj_scalar, obj_negate = batch_util.objective_scalar(batch)
          
              def fn(trial_flags):
                  run = batch_util.run_trial(batch, trial_flags)
                  obj_value = index.scalars(run).get(obj_scalar)
                  if obj_value is None:
                      log.error(
                          "Trial %s did not log objective '%s' - cannot use to optimize",
                          run.id,
                          obj_scalar,
                      )
                      return {
                          "status": hyperopt.STATUS_FAIL,
                      }

The result is the objective specified in guild run — or loss by default.

Minimize the Objective

Call fmin to minimize the objective:

github.com

guildai/guildai/blob/main/examples/hyperopt/tpe.py#L32


      
          batch_util.init_logging()
          batch = batch_util.batch_run()
          batch_flags = batch.get("flags")
          max_trials = batch.get("max_trials") or DEFAULT_MAX_TRIALS
          random_seed = batch.get("random_seed")
          trials = hyperopt.Trials()
          try:
              hyperopt.fmin(
                  fn=_objective_fn(batch),
                  space=_space_for_flags(batch),
                  algo=_tpe_suggest,
                  max_evals=max_trials,
                  show_progressbar=False,
                  rstate=np.random.RandomState(random_seed),
                  trials=trials,
              )
          except hyperopt.exceptions.AllTrialsFailed:
              pass
          else:
              _label_best(trials.best_trial)

Configure TPE Algorithm

The value for algo uses a local function that supports the TPE hyperparameters. The hyperparameters in this case are defined as module global variables. These are imported by Guild and can be customized using -Fo options with guild run.

github.com

guildai/guildai/blob/main/examples/hyperopt/tpe.py#L109


      
                  else:
                      return space
          
          
          def _tpe_suggest(new_ids, domain, trials, seed):
              return hyperopt.tpe.suggest(
                  new_ids,
                  domain,
                  trials,
                  seed,
                  prior_weight=prior_weight,
                  n_startup_jobs=startup_jobs,
                  n_EI_candidates=EI_candidates,
                  gamma=gamma,
              )
          
          
          def _label_best(trial):
              run = trial["result"]["run"]
              log.info("Labeling best run %s", run.id)
              run.write_attr("label", "best %s" % run.get("label"))

Perform Summary Actions

This implementation labels the “best” run. Run any number of summary actions in your custom optimzer.

github.com

guildai/guildai/blob/main/examples/hyperopt/tpe.py#L116


      
                  domain,
                  trials,
                  seed,
                  prior_weight=prior_weight,
                  n_startup_jobs=startup_jobs,
                  n_EI_candidates=EI_candidates,
                  gamma=gamma,
              )
          
          
          def _label_best(trial):
              run = trial["result"]["run"]
              log.info("Labeling best run %s", run.id)
              run.write_attr("label", "best %s" % run.get("label"))
          
          
          if __name__ == "__main__":
              main()

Topic		Replies	Views
How to use Tune (ray) with Guild General	4	1075	September 9, 2022
Hyperopt TPE vs Skopt Gaussian Processes Troubleshooting	0	1413	July 11, 2020
Hyperparameter Optimization Concepts	0	3252	June 12, 2020
Optimizers Reference Reference	0	2100	June 7, 2020
Get Started: Optimize a Model Get Started	1	8858	March 23, 2021

Overview

Implementation

Read Batch Settings

Convert Prototype Flags to Search Space

Function to Minimize

Minimize the Objective

Configure TPE Algorithm

Perform Summary Actions

Related topics