Guild Stage-Trials Rerun on Error

For my academic project, I use as a lightweight job scheduler on our GPU cluster node.
Following some of the tutorials on, I am currently using several queues to split tasks to different GPUs on the node.

guild run --background --yes queue gpus=i

then I simply submit tasks using guild run --stage-trials ....

(This works amazingly well for other users that need such a functionality!)

The problem: The cluster is automatically re-balancing loads, so the node is subject to restarts.
This would be fine because runs show up as “error” (as far as I understand due to PID mismatch on the restarted node, since the same file system is mounted again). However I am currently unable to reset runs from state “error” to “staged”, preventing me from using it in fully automated manner.

I tried to use the run restart functionality, but it seems incompatible with stage-trials.

Some examples of what I tried:

guild run --restart --stage <run>
guild run --proto --stage <run>
guild run --stage-trials `guild select -Se :`

PS: Thank you for this amazing project!