I am trying to learn a recipe (a sequence of hyperparameters) to train a model. In essence, I would like to have the model requiring itself in a fashion like this:
op:
main: XX
flags: [some flags]
requires:
- operation: op
select: [saved model]
pipeline:
steps:
- op [flags setting 1]
- op [flags setting 2]
But obviously, the first op won’t have an existing run to require.
The only work around I can think of is to have the flag isolate-runs set to no and manually create the first run, but this ain’t pretty. I wonder if there’s an optional option for requires.
requires:
- operations: op
select: [saved model]
optional: yes
I tried setting fail-if-empty to no, but it didn’t help
The fail-if-empty is working as designed here. That is not what you’re looking for semantically (it applies to what’s selected from a resource, not to make the resource itself optional - subtle maybe but I think it’s a different topic).
We want optional for a requirement/resource. This was requested a while back so we need to get it in.
Not a bug - the fail-on-empty is a different meaning altogether (it applies when we know what the run is but there are no files selected for it). We don’t want to overload this to mean “this dependency is optional”.
This is related to this topic:
I think optional here is what we want. This applies to the required resource itself, not what gets selected from it.
The proposal is to introduce a fail-if-unresolved that can be set to no/false to allow a run to continue when a requirement cannot be satisfied. This is a bit a tongue-twister relative to the more simple optional attribute, but it follows the convention in place by fail-if-empty.
I don’t have a strong opinion about the name and can be swayed! The important part is that this works.
This will be available in the next release. I’ll update here when it’s available.