Hello and welcome!
I’m guessing the file you’re looking for is being picked up as source code and is located under .guild/sourcecode under the run directory.
For example, I would expect this to not show the file:
guild ls
I would expect it to appear in this case:
guild ls --sourcecode
There are a number of ways you might address this:
- Specify an absolute path to the config file rather than a relative path.
- Modify your script to look under
.guild/sourcecode when it gets a relative path for this value.
- Modify the Guild file to store your source code under the run directory root.
- Modify the Guild file to make all possible config files available as a dependency.
- Get real fancy and use a flag reference in your dependency spec to resolve only the file needed.
Haha, sorry - that’s lot of approaches. Let me fill in each. You have an important question and the answers here are generally helpful.
Option 1 - absolute paths
This is a quick-and-dirty method that takes the relative path problem off the table. Use this to get things working ASAP. It’s not a good solution because it’s totally surprising and unintuitive.
Option 2 - look for config files under source code dir
Any time you’re changing your code to accommodate Guild it’s an anti-pattern. In some cases Guild may be missing an important feature and there’s no other simple way to fix the problem. In this case, I think there are better options.
Option 3 - Save source code in the run root
You can do this with this config:
train:
sourcecode:
dest: .
This isn’t bad but it now floods your run directory with all your source code files. Personally I like to have the source code out-of-view and .guild/sourcecode is a good spot. This lets you focus on your run-generated files and required files for each run.
Option 4 - Include all possible config files as dependencies
This is the option that I prefer as it’s not terribly complicated and is pretty good spelling of what you want I think.
In your Guild file, it’d look like this:
train:
requires:
- file: config_.*\.json
Note the pattern is a regular expression and not a glob wildcard. Sorry about that! There’s an issue that talks about changing this to globs by default.
This is slightly brute force as it copies all possible config files rather than just the one you want. Still, it’s direct and avoids the head-scratchingly strange pattern used in the last option.
Option 5 - use a flag ref in your dependency spec
This approach uses a flag to specify the config file—or part of the filename—and uses that flag value in a dependency spec.
train:
flags-import: all
flags-import-skip:
- config_filepath
flags:
config:
description: Config to use in train
required: yes
choices: [...] # optionally make legal values available
arg-name: config_filepath
requires:
- file: ${config}
You could spell the file pattern as ${config}_h5.json and make it more of a name than a filename.
You want to skip importing config_filepath because that exposes the user to the problem you’re facing now.
This method is more complicated but I think it’s nice to insulate the user from file path details. I’m not officially recommending this because option 4 is so much simpler. But this is probably the approach I’d take in a serious effort to Guildify a project.
Clear as mud?