Specifying different target types for the same operation

Troubleshooting
1

I am trying to link and copy different files from the same operation resource in my guild stage. It seems that specifying an operation multiple times only results in the last operation being used. Here is an example that matches what I’m trying to do where a operation train is used as a resource. The file model.pth is linked while config.json is copied.

test:
  exec: "python test.py"
  requires:
    - operation: train
      select:
        - file: model.pth
          target-type: link
        - file: config.json
          target-type: copy

Is there any way to achieve this kind of functionality? Thanks

2

You need to do it through a pipeline, or manually assign the run ID yourself.

op1:
  exec: func_1
  flag: 
    a: 1E5
    b: 1E-5
  requires: ...

op2:
  exec: func_2
  flag: 
    b: 1E-5
    c: ''
  require: 
    - operation: op1
      select: '[folder or file generated by op1]'

pipeline:
  flag:
    a: 1E5
    b: 1E-5
    c: ''
  steps:
    - op1 a=${a} b=${b}
    - op2 b=${b} c=${c}

This is covered in https://my.guild.ai/t/pipelines/163.

Also, note that your exec attribute should assign the moduel/function name rather than the command to execute for python scripts. This lead to funny behavior if you have multiple argparsers in the script.

1 Like
3

Revisiting this problem now there’s another solution:

# Generates files you need both to link and copy
op1: 
  exec: func_1
  
# Next step
op2:
  exec: func_2
  requires:
    - copy-op1
    - link-op1

resources:
  copy-op1:
    - operation: op1
      select: 
      - [REGEX]
      target-type: 'copy'
      target-path: 'Copy_Op1'
  
  link-op1:
    - operation: op1
      select:
      - [REGEX]
      target-type: 'link'
      target-path: 'Links_Op1'