I read in the documentation that it is possible to use manual/grid search and random search. In docs, it says that I use random search if I use uniform function in at least one variable. But is it possible to use a random search on a grid?
For example if I want to run guild run train x=[1,2,3,4], but want that method randomly choose 2 models (e.g. 2 and 4). Is it possible to do that?
Hello and welcome! I thought it was possible to do this and then went to test this and it’s not 
There’s a problem with the default batch support (the logic that implements grid support) - it doesn’t detect distributions like uniform[-10:10] and so treats those values as strings rather than functions to draw samples from.
Could I trouble you to open an issue for this problem? I’ll create a resolution project for it and see if there are some work-arounds. 0.7 is currently frozen for a pending release but I can create a branch with a fix for this.
@garrett, I have opened the issues with the same question. Is it possible than to set uniform only for those 4 values and set the number of models to test to 2?
Terrific - thank you!
In re-reading you issue I think I initially misunderstood what you’re asking for.
If you want to run two trials and select a single flag value at random, Guild would consider this a random search. You can do this by running:
guild run train x=[1,2,3,4] -o random
You can always specify an optimizer using -o or --optimizer to override Guild’s automation detection behavior.
When the random optimizer sees a list of values for a flag, it treats that list as a category dimension and selects randomly from the list of values.
What Guild currently does not support — and this is what I had originally thought you were asking about — is to integrate random selection with a true grid search. Something like this:
guild run x=[1:4] y=[5,6] z=[7,8] -o +
Does NOT work as of 0.7.0.rc11
Guild sees the value [1:4] for x in this case and treats it as a string. Your script gets the literal value rather than a random section from that range. I have a branch grid-support-flag-functions where this is fixed but it’s not released yet.
I think also the following command should be a grid search that generates four trials and where x is randomly selected for each of the four trials:
guild run x=uniform[1,2,3,4] y=[5,6] z=[7,8]
Does NOT work as of 0.7.0.rc11
This is a sensible enhancement we could make in the grid-support-flag-functions work.
Actually, there’s an even easier way to do what you’re looking for. Specify your list of possible values and use --max-trials to limit the runs. Guild selects from available trials randomly.
guild run train x=[1,2,3,4] -m 2
This generates two trials where values for x are selected from the list. In this case the set of trials will be unique. Guild’s algorithm here removes trials from the grid search at random until the max number is reached.
So, in nutshell, this works:
guild run x=[1,2,3,4,] -o random
and this doesn’t work
guild run x=[1:4]
How can I choose only 2 models at random? Something like (this is probably wrong):
guild run x=[1,2,3,4,] -o random 2
I thought -m doesn’t choose randomly. If it does, this is the solution for my problem
The -m option is short for --max-trials. This limits the number of trials generated for a command.
In the case of a grid search where the number of generated trials is greater than max trials, Guild will randomly remove a number of trials to ensure that no more than max trials are generted.
Based on what I think you’re asking for, this is what I’d use:
guild run x=[1,2,3,4] -m 2
This is a grid search because there’s a list of values. Guild wants to run four trials here. But because you’re limiting the number to 2, Guild randomly removed two of the four trials.
If you want this to be deterministic (for a given Python version) use --random-seed.
You can preview the trials using the --print-trials option.
Does this work wtih grid, like
guild run x=[1:4] -m 2
?
That command will use the random optimizer. You can see what Guild chooses in the preview message.
guild run train x=[1:4] -m 2
You are about to run train with random search (max 2 trials)
noise: 0.1
x: [1:4]
Continue? (Y/n)
This will generate two trials, selecting values for x at random over the uniform distribution of integers from 1 to 4 inclusive.
There is no assurance in this case that you will get unique values for x. You can run this command with --print-trials to see what Guild generates.
The command I suggest above does ensure that you’ll get unique values for x.
Ok ,thanks, I will experiment now with above approaches.
@garrett, sorry, I have one more question. What if I use csv file with flags and than try to run batch with csv file. Can I than also use random option? I haven’t tried to use batch file yet but I plan to use it in the future.
Batch trials work the same as grid search. In fact you can combine batch file trials with flags on the command line to expand the batch trials in a search.
The --max-trials option also works the same. If you specify a max value that’s less than the number of generated trials, Guild prunes the list down to the max randomly.