In a batched trial pipeline see 196, how can we run as fast as possible parallelizing everything we can ?
Example Pipeline:
operation_1: # has no dependencies
operation_2:
requires:
- operation: operation_1
flags-dest: globals
flags-import:
- some_param
operation_3:
requires:
- operation: operation_2
flags-dest: globals
flags-import:
- some_other_param
pipeline:
steps:
- run: operation_1
- run: operation_2
flags:
some_param: [a, b]
- run: operation_3
flags:
some_other_param: [1, 2, 3]
# create 6 queues as we have 6 batch trials that can be done in parallel (a1, a2, a3, b1, b2, b3, c1, c2, c3)
# run this command 6 times
guild run queue --background
guild run pipeline --stage
For some reason this leads to out of sequence events happening, like operation 2 being run before operation 1 resulting in an error. Is this the correct way to do this?
Further, is there a good way of timing this to sanity check parallel works faster i.e time batched trials run?