Dependecies Problem

Dear all,
I am new to Guild.ai
I have a train file called train.py that needs as input a parameter (using argparse) that is called config_filepath .
I am trying over and over to run:
guild run train.py config_filepath=config_thyroid_segnet_multi_h5.json

but nothing works. Everytime I get the following message:
FileNotFoundError: [Errno 2] No such file or directory: ‘config_thyroid_segnet_multi_h5.json’

The funny part is that if I check in the runs directory (performing guild ls) the file is there. What am I doing wrong? I tried also to create a guild.yml file like following:
train.py:
description: Say hello to my friends
main: train
default: yes
flags-import:
- config_filepath

And alternatively to create a guild.yml with the tag requires.

Nothing works…

Thanks in advance for your support

Hello and welcome!

I’m guessing the file you’re looking for is being picked up as source code and is located under .guild/sourcecode under the run directory.

For example, I would expect this to not show the file:

guild ls

I would expect it to appear in this case:

guild ls --sourcecode

There are a number of ways you might address this:

  1. Specify an absolute path to the config file rather than a relative path.
  2. Modify your script to look under .guild/sourcecode when it gets a relative path for this value.
  3. Modify the Guild file to store your source code under the run directory root.
  4. Modify the Guild file to make all possible config files available as a dependency.
  5. Get real fancy and use a flag reference in your dependency spec to resolve only the file needed.

Haha, sorry - that’s lot of approaches. Let me fill in each. You have an important question and the answers here are generally helpful.

Option 1 - absolute paths

This is a quick-and-dirty method that takes the relative path problem off the table. Use this to get things working ASAP. It’s not a good solution because it’s totally surprising and unintuitive.

Option 2 - look for config files under source code dir

Any time you’re changing your code to accommodate Guild it’s an anti-pattern. In some cases Guild may be missing an important feature and there’s no other simple way to fix the problem. In this case, I think there are better options.

Option 3 - Save source code in the run root

You can do this with this config:

train:
  sourcecode:
    dest: .

This isn’t bad but it now floods your run directory with all your source code files. Personally I like to have the source code out-of-view and .guild/sourcecode is a good spot. This lets you focus on your run-generated files and required files for each run.

Option 4 - Include all possible config files as dependencies

This is the option that I prefer as it’s not terribly complicated and is pretty good spelling of what you want I think.

In your Guild file, it’d look like this:

train:
  requires:
    - file: config_.*\.json

Note the pattern is a regular expression and not a glob wildcard. Sorry about that! There’s an issue that talks about changing this to globs by default.

This is slightly brute force as it copies all possible config files rather than just the one you want. Still, it’s direct and avoids the head-scratchingly strange pattern used in the last option.

Option 5 - use a flag ref in your dependency spec

This approach uses a flag to specify the config file—or part of the filename—and uses that flag value in a dependency spec.

train:
  flags-import: all
  flags-import-skip:
    - config_filepath
  flags:
    config:
      description: Config to use in train
      required: yes
      choices: [...]  # optionally make legal values available
      arg-name: config_filepath
  requires:
    - file: ${config}

You could spell the file pattern as ${config}_h5.json and make it more of a name than a filename.

You want to skip importing config_filepath because that exposes the user to the problem you’re facing now.

This method is more complicated but I think it’s nice to insulate the user from file path details. I’m not officially recommending this because option 4 is so much simpler. But this is probably the approach I’d take in a serious effort to Guildify a project.

Clear as mud?

Thank you very much for your answer. I actually tried your best solution and now the problem that I face in Windows 10 Home Edition is:
guild: run failed because a dependency was not met: unable to link to dependency source: [Errno 1] You do not have sufficient privilege to perform this operation.

The only solution that works is the absolute path, but it´s a solution that I don´t want to use.

If you open your command line app as Administrator this error goes away.

Alternatively, add target-type: copy to the file dependency:

op:
  requires:
    - file: ...
      target-type: copy

This tells Guild to copy the files rather than link to them so you won’t need that privilege in Windows.

Thank you very much. It works if I input as file directly the name of the file (and I have to repeat the target-type: copy after each of them). The regular expression that you sent me, in spite being correct, doesn´t work:
guild: run failed because a dependency was not met: could not resolve 'file:config_..json’ in file:config_..json resource: cannot find source file 'config_.*.json

train:
  description: training script
  requires:
    - file: config_.*\.json
      target-type: copy
    - file: .*\.schema
      target-type: copy
  flags-import: all

I gave you totally wrong semantics there! Here’s what you want:

train:
  description: training script
  requires:
    - file: .
      select: config_.*\.json
      target-type: copy
      name: config
    - file: .
      select: .*\.schema
      target-type: copy
      name: schema
  flags-import: all

This spelling is maybe a bit surprising but here’s what’s going on. You’re not resolving a single file but a set of files coming from a container. Guild lets you resolve files from directories or archives. In this case the files you want are in the project directory (.) you select them using the pattern in select. This technique can be applied to selecting files from a tar or zip file as well. That’s why the spelling is perhaps a little odd.

I added name there to help the progress read better. Otherwise Guild makes up a name using the file spec.

Amazing, thank you very much, this is exactly what I was searching for. Your library is amazing, I would just suggest to improve the documentation because I would have never reached your solution without you.
Thank you again