Guild runs very slow

Troubleshooting
1

Similarly to what has been described in this thread, running the following command results in hanging for a while before we see any results:

guild run train

The Guild file contains the following:

train:
  main: script

The script is simply:

print("Hello, world!")

I believe the hanging is related to the fact that I have a data folder which contains thousands of files each corresponding to a sample of the dataset I’m using in my experiments. This is because:

  1. When using the --debug flag, Guild hangs just after copying one .csv file within the data folder (none of the thousands of sample files are copied to the target directory, though).
  2. When I remove the data folder, the hanging stops.

My questions are:

  1. Is this a bug? If so, how can I circumvent this issue?
  2. How am I supposed to make my dataset accessible by the training script, since it is run from a totally different directory and there are thousands of files it needs to access?
2

Apologies for the late reply here! I’m looking into this now. I suspect it’s Guild’s scanning for source code files to copy but I want to confirm this before getting the correct config to you.

As for getting your script access to the data dir, you’d do that using a file dependency in your Guild file.

train:
  main: script
  requires:
    - file: data
      target-type: link

The target type ‘link’ there tells Guild NOT to copy all those files per run but just to link to the directory. If you want to copy, omit targe-type or use the value copy (the default behavior since 0.8).

1 Like