Guild doesn't copy module to new source code location

Troubleshooting
1

When I use the command guild run train,py I get the error Error: can't important module_1. When I look in the folder to which guildai copies the source code after I call guild run, I can see that module_1 was indeed not copied. There is nothing special about that module_1 (normal code files). How can I debug this issue further?

Here is my folder structure:

Folder structure:

guild.yml
train.py
module_1
module_2
module_3

guild.yml

train:
  description: Training script
  main: train
  # sourcecode:
  #   - '*.py'
  flags-dest: global:params
  flags-import: all
  flags:
    ## general
    gpu:
      description:
      default: 0
    seed:
      description:
      default: 0
...

Thanks for your help!

2

Hi @jan_r, I’m sorry you’re running into issues here.

Guild should be picking up any Python modules from module_* dirs and copying them to the .guild/sourcecode directory in the run. Note that Guild does not attempt to duplicate your project structure. Source code files do not appear in the run directory root, at least by default.

You can view the list of source code files for a run this way:

guild ls --sourcecode  # shows source code files for latest run

You can test the source code copy logic ahead of time for a run this way:

guild run train --test-sourcecode

This gives you a detailed list of files that are copied and those that are skipped based on the sourcecode config for the operation.

Note that guild run train.py runs the the script directly. In this case, because your operation is named train (no .py extension), Guild does not use the config in guild.yml. You need to use guild run train, which uses the operation name in guild.yml.

3

One thought here — the module_* dirs must each have a __init__.py file to indicate they’re a package. In my tests, when I removed that file I get the error message you show above.

4

That’s good advice thanks! Unfortunately, they relevant .py files in my module which are not copied don’t appear in the list of skipped files nor in the list of copied files. The module has a __init__.py file so it’s a proper module.

Do you have any other tricks I could use for debugging?

5

Are the Python source code files stored under a linked directory? What OS is this? Thanks!