Parallel batch trial pipeline

In a batched trial pipeline see 196, how can we run as fast as possible parallelizing everything we can ?

Example Pipeline:

operation_1: # has no dependencies

        - operation: operation_1
    flags-dest: globals
        - some_param

        - operation: operation_2
    flags-dest: globals
        - some_other_param

    - run: operation_1
    - run: operation_2
        some_param: [a, b]
    - run: operation_3
        some_other_param: [1, 2, 3]
# create 6 queues as we have 6 batch trials that can be done in parallel (a1, a2, a3, b1, b2, b3, c1, c2, c3)
# run this command 6 times
guild run queue --background 

guild run pipeline --stage

For some reason this leads to out of sequence events happening, like operation 2 being run before operation 1 resulting in an error. Is this the correct way to do this?

Further, is there a good way of timing this to sanity check parallel works faster i.e time batched trials run?

Don’t use staged runs for this. Just run your pipelines in the background. At some point Guild will be optimized for parallel runs and you won’t have to think about this (as you say - we want to run everything as fast as possible). But at the moment, you need to use parallel OS processes to manage parallel runs.