Self requiring operation

I am trying to learn a recipe (a sequence of hyperparameters) to train a model. In essence, I would like to have the model requiring itself in a fashion like this:

op:
  main: XX
  flags: [some flags]
  requires:
    - operation: op
      select: [saved model]

pipeline:
  steps:
    - op [flags setting 1]
    - op [flags setting 2]

But obviously, the first op won’t have an existing run to require.

The only work around I can think of is to have the flag isolate-runs set to no and manually create the first run, but this ain’t pretty. I wonder if there’s an optional option for requires.

requires: 
  - operations: op
    select: [saved model]
    optional: yes

I tried setting fail-if-empty to no, but it didn’t help

I think there’s a bug in play - fail-if-empty should work in this case. Stay tuned, we should be able to get a fix out for this quickly.

1 Like

The fail-if-empty is working as designed here. That is not what you’re looking for semantically (it applies to what’s selected from a resource, not to make the resource itself optional - subtle maybe but I think it’s a different topic).

We want optional for a requirement/resource. This was requested a while back so we need to get it in.

1 Like

Hi Garrett, thanks for promptly looking into this! I think i can live with this for a while, I managed to workout using a trick like this:

op:
  ...

op_looped:
  ... # same as op
  requires:
    - operation: op
      select: [Saved_model]

train-recipe:
  steps:
    - op [flags 1]
    - op_looped [flags 2]
    - op_looped [flags 3]

Thanks for your insights!

1 Like

Not a bug - the fail-on-empty is a different meaning altogether (it applies when we know what the run is but there are no files selected for it). We don’t want to overload this to mean “this dependency is optional”.

This is related to this topic:

I think optional here is what we want. This applies to the required resource itself, not what gets selected from it.

Agreed. I do recon my use case is not very typical. Thank you for looking into it anyways.