By default the scheduler will simply run the staged runs concurrently, up to the number of workers specified. It doesn’t assign runs to GPUs. This is a good feature idea, but the initial pass doesn’t do this.
Instead, you need to assign runs to GPUs explicitly. This uses Dask resources to limit runs, as well as configuring each run for the target GPU.
The real target is to support this with just the run command—so you don’t need to mess with schedulers. But this first pass introduced concurrency with Dask distributed and so exposes the resource based scheduling that Dask offers.
If you have any questions about the steps in Scheduling Runs on Specific GPUs please feel free to followup here. We’ll look for ways to clean this up, either via better docs or tweaks to the code.