In a batched trial pipeline see 196, how can we run as fast as possible parallelizing everything we can ?
operation_1: # has no dependencies operation_2: requires: - operation: operation_1 flags-dest: globals flags-import: - some_param operation_3: requires: - operation: operation_2 flags-dest: globals flags-import: - some_other_param pipeline: steps: - run: operation_1 - run: operation_2 flags: some_param: [a, b] - run: operation_3 flags: some_other_param: [1, 2, 3]
# create 6 queues as we have 6 batch trials that can be done in parallel (a1, a2, a3, b1, b2, b3, c1, c2, c3) # run this command 6 times guild run queue --background guild run pipeline --stage
For some reason this leads to out of sequence events happening, like operation 2 being run before operation 1 resulting in an error. Is this the correct way to do this?
Further, is there a good way of timing this to sanity check parallel works faster i.e time batched trials run?