# Get Started: Optimize a Model

## Overview

In the previous section you use Guild to run `train.py` — a sample training script. In this section, you run the same script with different hyperparameter values to find lower values of `loss`. This process is known as hyperparameter optimization, or hyperparameter tuning.

Guild supports hyperparameter optimization using different search methods:

Here’s the loss function used in `train.py`:

``````loss = (np.sin(5 * x) * (1 - np.tanh(x ** 2)) + np.random.randn() * noise)
``````

The relationship between `x` and `loss` is plotted below. Optimal values for `x` are around `-0.3`.

Plot of `x` on the horizontal axis to `loss` on the vertical axis (image by Gilles Louppe, Manoj Kumar)

In a real scenario, we don’t know optimal hyperparameter values. We need to search for them.

## Manual Search

When experimenting with hyperparameters, it’s often useful to start with values based on prior experience or intuition.

Guild lets you run multiple trials in a batch by specifying them as a list in the form `[VAL1,VAL2,…,VALN]`.

Run three trials of `train.py` using different values for `x`:

``````guild run train.py x='[-1,0,1]'
``````
``````You are about to run train.py as a batch (3 trials)
noise: 0.1
x: [-1, 0, 1]
Continue? (Y/n)
``````

Important The single quotes used for the argument `[-1,0,1]` are required if you’re using Z shell (zsh). This is the default shell used in macOS. If you’re using Bash or Bourne shells, you can omit them provided the list argument does not contain spaces.

Press Enter to start the batch.

Guild runs `train.py` three times, once for each specified value of `x`.

Show the runs with `guild runs`:

``````guild runs
``````
``````[1:1933bdcb]  train.py   2020-01-14 09:38:15  completed  noise=0.1 x=1
[2:83dc048d]  train.py   2020-01-14 09:38:14  completed  noise=0.1 x=0
[3:468bb240]  train.py   2020-01-14 09:38:14  completed  noise=0.1 x=-1
[4:bfcff413]  train.py+  2020-01-14 09:38:13  completed
[5:68f4da74]  train.py   2020-01-14 08:42:54  completed  noise=0.1 x=0.1
``````

Runs 1 through 3 are trial runs generated by the command. Run 4 is a batch run. Batch runs generate trials and are denoted using `+` in their name.

### Compare Runs

Compare `loss` across runs using `guild compare`:

``````guild compare --min loss
``````

Guild starts an interactive console application that lets you browse experiment results. Runs with lower `loss` appear at the top because `--min loss` is specified in the command. Use arrow keys to navigate. Press 1 to sort by the current column in ascending order or 2 to sort in descending order. Press ? for a list of supported commands.

Compare experiment results — press ? for a list of commands; press q to exit

Press q to exit Guild Compare.

Highlight Guild Compare is a convenient way to review run results. It works across remote terminal connections as well as local environments and doesn’t require a web browser or port configuration.

Next, run four trials — one for for each unique combination of flag values:

``````guild run train.py x=[-0.5,0.5] noise=[0.1,0.2]
``````
``````You are about to run train.py as a batch (4 trials)
noise: [0.1, 0.2]
x: [-0.5, 0.5]
Continue? (Y/n)
``````

Press Enter to start the batch.

Show the top-3 best runs:

``````guild compare --table --min loss --top 3
``````

The `--table` option tells Guild to show results without running in interactive mode. The `--top` option tells Guild to show only the top-N runs based on the sort order.

Runs where `x` is `-0.5` should have the lowest values for `loss`. This is consistent with our expectation from the plot above.

### View Runs in TensorBoard

Guild integrates TensorBoard for exploring runs. Use `guild tensorboard` to view current runs:

``````guild tensorboard
``````

Guild starts TensorBoard and opens a new tab in your browser.

Select the HParams tab and then select Parallel Coordinates View.

Compare runs using Parallel Coordinates View

The Parallel Coordinates View highlights runs that perform better along various axes. Click-and-drag along the `loss` axis to highlight runs with the lowest values. Note that runs where `x` is `-0.5` are highlighted.

Highlight Guild automatically generates HParam summaries from Guild runs to simply the process of comparing runs in TensorBoard.

## Grid Search

Grid search is a systematic search across a subset of hyperparameter space. Guild supports a special sequence function syntax for specifying value ranges. Ranges are specified using the format `FUNCTION[START:END:STEP_OR_COUNT]` where `FUNCTION` is the type of sequence and `START` and `END` mark the start and end of the sequence respectively. `STEP_OR_COUNT` is the range step or value count depending on the function used.

Use the `linspace` flag function to run four trials where `x` is evenly spaced between `-0.6` and `0.6`:

``````guild run train.py x=linspace[-0.6:0.6:4]
``````
``````You are about to run train.py as a batch (max 20 trials, minimize loss)
noise: 0.1
x: [-0.6, -0.2, 0.2, 0.6]
Continue? (Y/n)
``````

Guild expands the function `linspace[-0.6:0.6:4]` to the list `[-0.6, -0.2, 0.2, 0.6]`.

Press Enter to start the batch.

Guild runs trials for each value of `x` generated by the `linspace` function.

Tip Use sequence functions for multiple flags to expand a grid search to the Cartesian product of the flag value sets.

## Random Search

Guild supports random search over a both uniform and log-uniform distributions.

Search space is specified by special search space functions, which include `uniform` and `loguniform`. The `uniform` function name may be omitted.

To search over a uniformly distributed range of values, specify a flag value in the format `[MIN:MAX]`. By default, Guild runs 20 trials using randomly chosen values within the specified range. Use `--max-trials` to specify a different limit.

Start a random search over `x` with 5 trials:

``````guild run train.py x=[-2.0:2.0] --max-trials 5
``````
``````You are about to run train.py with random search (max 5 trials)
noise: 0.1
x: [-2.0:2.0]
Continue? (Y/n)
``````

Press Enter to start the batch.

Guild runs `train.py` five times using randomly sampled values for `x` from `-2.0` to `2.0`.

## Bayesian Optimization

Bayesian optimization uses light-weight probabilistic models to suggest hyperparameter values believed to improve a future result based on previous results.

By default, Guild attempts to minimize `loss` as an objective. The `train.py` script logs `loss` as a scalar. If the script uses a different objective, specify the scalar using `--minimize` or `--maximize` options with `guild run`.

Run 10 trials using the `gp` optimizer, which uses Bayesian optimization with Gaussian processes, and search over values of `x` between `-2.0` and `2.0`:

``````guild run train.py x=[-2.0:2.0] --optimizer gp --max-trials 10
``````
``````You are about to run train.py with 'skopt:gp' optimizer (max 10 trials, minimize loss)
noise: 0.1
x: [-2.0:2.0]
Optimizer flags:
acq-func: gp_hedge
kappa: 1.96
noise: gaussian
random-starts: 3
xi: 0.05
Continue? (Y/n)
``````

Press Enter to start the batch.

The `gp` optimizer seeds the batch with three random starts. After the third random start, the optimizer uses previous results to suggest candidates for `x` between `-2.0` and `2.0` to minimize `loss`.

For list of available optimizes, see Optimizers Reference.

### Restart Batch to Continue Optimization

Guild optimizers use previous results from trials of the same batch. If you want to continue a search for hyperparameters using previous results, restart the batch run.

To restart a run, specify `--restart` with `guild run`.

To get the run ID of the `gp` batch, run:

``````guild select -Fo train.py+gp
``````

Guild prints the ID of the last `gp` batch run. Copy this value for the next command.

Restart the batch to generate another ten trials, replacing `BATCH_RUN_ID` below with ID from the previous command:

``````guild run -Fo random-starts=0 --restart BATCH_RUN_ID
``````
``````You are about to start ... (train.py) with 'skopt:gp' optimizer (max 10 trials)
noise: 0.1
x: [-2.0:2.0]
Optimizer flags:
acq-func: gp_hedge
kappa: 1.96
noise: gaussian
random-starts: 0
xi: 0.05
Continue? (Y/n)
``````

Tip You can type the first few characters of the batch run ID rather than use the full ID if those characters uniquely identify the run.

Tip If you’re running Linux, macOS, or another POSIX environment, you can use command substitution and `guild select` to specify a run ID argument. For example, `guild run --restart \$(guild select -Fo train.py+gp)` restarts the latest `train.py+gp`.

Press Enter to restart the batch.

Guild generates another ten trials for the batch. Guild uses the previous trials generated earlier as inputs to the optimization.

The option `-Fo random-starts=0` is an optimizer flag that sets the number of random starts to `0`.

### Evaluate Bayesian Optimization Results

Run `guild compare` for the last 20 runs to evaluate the results of the Bayesian optimization:

``````guild compare :20
``````

By default, Guild orders runs by start time with most recent runs listed first. Note values for `x` suggested by the `gp` optimizer. If the optimizer succeeds in finding values for `x` that minimize `loss`, you should see more values around `-0.3` toward the top of the list. These are the most recent runs and benefit from more trial data.

Press q to exit Guild Compare.

You can also use TensorBoard to evaluate optimization results.

Start TensorBoard for the last 20 runs:

``````guild tensorboard :20
``````

Guild starts TensorBoard and opens a new browser tab.

Select the HParams tag and then select Scatter Plot Matrix View. TensorBoard shows scatter plots of hyperparameters and metrics.

In the left side panel, deselect the following hyperparameters:

• `noise`
• `sourcecode`

In the same panel, deselect the following metrics:

• `time`

TensorBoard displays a plot of `loss` against `x`. Each point on the plot represents a trial. Given the known relationship between `loss` and `x` (see plot above), the Bayesian optimizer should spend more time exploring values for `x` around `-0.3`. This will appear as a cluster of trials along the bottom of the plot between `-0.4` and `-0.2`.

Plot `loss` against `x` to evaluate the Bayesian optimization results. Effective optimization spends more time exploring values of `x` around `-0.3`, shown as a cluster of trials along the bottom axis.

## Summary

In this section, you use various techniques to run `train.py`.

Highlights

• Guild supports a variety of optimization methods including grid search, random search, and Bayesian optimization.
• Use Guild Compare and TensorBoard to study optimization trials to help find the best hyperparameters for your models.

In the next section, you learn how to manage runs.

Next: Manage Runs

1 Like

A post was split to a new topic: Docs using -o option outdated