Re queues, I was going to write you here to say that queues will not start trials that are managed by a batch run — but I just tested this and that’s not the case. In 0.7.0 queues in fact do compete with batches that stage trials. This is a bug that will be fixed in 0.7.1!
You can use
--stage-trials to avoid these race conditions.
In looking over the Queues docs it looks like the info is covered but as you say it’s not clear how to use them. I wonder if an end-to-end example (e.g. a How To guide) would make things clearer.
--proto is similar to
--restart but it tells Guild to start a new run rather than restart an existing run. The run specified with
--proto is used as the prototype (or “template”) for the new run. So the operation, flags, source code, etc. are all used by default to start the new run. You can change flags and use
--force-sourcecode to use the current working code rather than the prototype’s.
The most common use case
--restart is when a run terminates early. You want to literally restart the run in place. Restartable runs need to check for interim save-points and re-initialize their states as needed on restart (Guild doesn’t do anything like that automatically). So the driver for using
--restart is that you have some saved stated in the run directory that you want to use.
--proto is less commonly used. I use it when I want to start a run using the flag values from a previous run but I’m too lazy to look up the values and re-type them. In that case I’d use
--force-sourcecode to make sure that my current source code is used. Another common case is to tweak some flag values but make sure the source code doesn’t change. In that case you do not use