Hi @nbren12 - sorry for the late reply!
Guild uses a file locking scheme to address contentions across queues. Locks are written to
$GUILD_HOME/locks — if this location is shared across processes (e.g. running on different nodes) queues should not attempt to start the same queued run. If they do that’s a bug. (This behavior is under test but for a locally mounted file system on a single node.)
Queues support a scheme for handling runs targeting a particular GPU, which lets you serialize runs per GPU. This could be used as a workaround per your request, but it has the side effect of setting
CUDA_VISIBLE_DEVICES per run, which is not something I’m comfortable recommending.
If I understand your request, Guild queues should support some method of selecting runs that match some criteria, otherwise they leave the staged run alone. E.g. tags or labels are logical candidates for associating a run with a queue.
Sorry again for getting back to you so late!