Self requiring operation

I am trying to learn a recipe (a sequence of hyperparameters) to train a model. In essence, I would like to have the model requiring itself in a fashion like this:

op:
  main: XX
  flags: [some flags]
  requires:
    - operation: op
      select: [saved model]

pipeline:
  steps:
    - op [flags setting 1]
    - op [flags setting 2]

But obviously, the first op won’t have an existing run to require.

The only work around I can think of is to have the flag isolate-runs set to no and manually create the first run, but this ain’t pretty. I wonder if there’s an optional option for requires.

requires: 
  - operations: op
    select: [saved model]
    optional: yes

I tried setting fail-if-empty to no, but it didn’t help

I think there’s a bug in play - fail-if-empty should work in this case. Stay tuned, we should be able to get a fix out for this quickly.

1 Like

The fail-if-empty is working as designed here. That is not what you’re looking for semantically (it applies to what’s selected from a resource, not to make the resource itself optional - subtle maybe but I think it’s a different topic).

We want optional for a requirement/resource. This was requested a while back so we need to get it in.

1 Like

Hi Garrett, thanks for promptly looking into this! I think i can live with this for a while, I managed to workout using a trick like this:

op:
  ...

op_looped:
  ... # same as op
  requires:
    - operation: op
      select: [Saved_model]

train-recipe:
  steps:
    - op [flags 1]
    - op_looped [flags 2]
    - op_looped [flags 3]

Thanks for your insights!

1 Like

Not a bug - the fail-on-empty is a different meaning altogether (it applies when we know what the run is but there are no files selected for it). We don’t want to overload this to mean “this dependency is optional”.

This is related to this topic:

I think optional here is what we want. This applies to the required resource itself, not what gets selected from it.

Agreed. I do recon my use case is not very typical. Thank you for looking into it anyways.

I’ve completed the proposal for this feature:

The proposal is to introduce a fail-if-unresolved that can be set to no/false to allow a run to continue when a requirement cannot be satisfied. This is a bit a tongue-twister relative to the more simple optional attribute, but it follows the convention in place by fail-if-empty.

I don’t have a strong opinion about the name and can be swayed! The important part is that this works.

This will be available in the next release. I’ll update here when it’s available.

This is implemented as optional - it’ll be available in 0.8.2.

1 Like