Can Guild take advantage of cached results?

General
1

Dear all,

I am a new user of Guild. I apologize if this question has already been answered; I could not find this information in the documentation.

I use Nextflow, a pipeline manager. One of my favorite features of Nextflow is the ability to cache results (via the -resume flag). This is useful when I ask nextflow to re-run a pipeline, but the input files, the parameters, and the script are unchanged. In this case, it won’t re-run the whole pipeline and just output the cached results. This takes off me the cognitive load of saving resources.

Is it possible to do the same on Guild? Specifically, my first project requires a good deal of parameter selection (the script and the input do not change, the specific parameters do). It would be great if Guild knew that it already explored a particular parameter combination and just skipped that computation.

Thank you for your help!

2

This is supported by Guild in a couple ways:

# guild.yml

data-prep: {}

train:
  requires:
    - operation: data-prep

The train operation in this case won’t run unless there’s an existing data-prep run to satisfy the requirement. By default Guild automatically selects the latest non-error data-prep run if one is available. You can alternative specify a run explicitly like this:

guild run train data-prep=<run ID>

Guild’s run command supports a --needed flag, which tells Guild to run the specified run only if an existing run doesn’t exist. In this case, Guild looks at the operation name and flag values. It does not look at changes to source code. In that respect, Guild’s reuse story isn’t as tight as Nextflow.

The --needed option can also be used in Guild’s pipeline support, so that if a particular step already exists, Guild skips it in the pipeline.

3

Thanks for your quick reply, garrett. As you say, the functionality is not exactly the same, but I can work around it.

4

You’re welcome! Please feel free to update here (or wherever) with questions/issues. If you do run into limitations, there may be a workaround, including using Nextflow itself to orchestrate Guild runs.