I was curious if there’s an idiomatic Guild way to run a batch of runs that are dependent on a batch of prior runs. To be concrete, suppose I run a batch of train runs. Also suppose I have an evaluation operation called eval_op that requires a train run (so the model checkpoint can be loaded in). It would be neat if something like this worked:
guild run eval_op train=[<run id 1>, <run id 2>, ...]
However, instead I have to type the runs out individually with each run id and also lose out on keeping it organized by batching them together. Full disclosure on context here: I did some hyperparameter optimization and I want to do some additional evaluation over each model.
Perhaps I missed something in the docs! Or if not, perhaps this could be a convenient feature?
The original syntax you specified is correct. In that case Guild will run eval_op once for each specified run ID. The only issue I see there is the use of spaces within the square brackets.
Something like this will do what you want, where the entries for the train flag list value are run IDs, or partial run IDs:
guild run eval_op train=[abc,def,fed,cba]
You would have to copy/paste the specific run IDs for this.
There’s a feature enhancement coming up that will let you run something like this:
Ah! I bet I did put a space. I’ll try again next time. I did end up instead using the queue solution combined with staging as you described. Thank you!