Parallel batch trial pipeline

In a batched trial pipeline see 196, how can we run as fast as possible parallelizing everything we can ?

Example Pipeline:

operation_1: # has no dependencies

operation_2: 
    requires:
        - operation: operation_1
    flags-dest: globals
    flags-import:
        - some_param

operation_3: 
    requires:
        - operation: operation_2
    flags-dest: globals
    flags-import:
        - some_other_param

pipeline:
  steps:
    - run: operation_1
    - run: operation_2
      flags:
        some_param: [a, b]
    - run: operation_3
      flags:
        some_other_param: [1, 2, 3]
# create 6 queues as we have 6 batch trials that can be done in parallel (a1, a2, a3, b1, b2, b3, c1, c2, c3)
# run this command 6 times
guild run queue --background 

guild run pipeline --stage

For some reason this leads to out of sequence events happening, like operation 2 being run before operation 1 resulting in an error. Is this the correct way to do this?

Further, is there a good way of timing this to sanity check parallel works faster i.e time batched trials run?

Don’t use staged runs for this. Just run your pipelines in the background. At some point Guild will be optimized for parallel runs and you won’t have to think about this (as you say - we want to run everything as fast as possible). But at the moment, you need to use parallel OS processes to manage parallel runs.