Import flags from dependency

Troubleshooting
1

I have the following guild yaml structure:

prepare-config:
    main: model.utils.prepare-config
    sourcecode: 
        root: /home/
        dest: .
        select: '*.py'

train:
    main: model.utils.run
    sourcecode: 
        root: /home/
        dest: .
        select: '*.py'
    requires:
      - operation: prepare-config
        select: prepare-config-output.json
        target-type: copy
        target-path: .
    flags-dest: config:prepare-config-output.json
    flags-import: all

In the run directory for the “train” operation I can see that the “prepare-config-output.json” file has been copied successfully. However when I perform “guild run train --help-op” I don’t see any of the parameters in “prepare-config-output.json”. I am even able to run “guild run train” successfully using the parameters in “prepare-config-output.json”, but it seems as though the flags are not imported and I cant use something like “guild run train.py lr=[0.1, 0.01, 0.001]” for example.

2

Hi @ap000 and welcome!

I’m not able to recreate this. I created an issue resolution doc to run your example. I’m running 0.7.4.rc1.

That’s a pre-release that you can install using pip install --pre --upgrade guildai.

3

My guess is your second operation never saw the json thrown by the first operation because you ran them separately.

To run two operations as a pipeline, you need to define a third operation “pipeline”:

op1:
  ...

op2:
 ...

pipeline:
  flags:
    op1flags: "default"
    op2flags: "default"
  steps: 
    - op1 op1flags=${op1flags}
    - op2 op2flags=${op2flags}

If you run op1 and op2 in two separate commands, they are strangers to each others unless you specify which run id of the upstream operation to use, like this:

guild run op1 ...
guild runs
# [1:xxxop1id] ...
guild run op2 op1=[xxxop1id]
1 Like