Tracking source code that is a python package

Troubleshooting
1

I’m very sorry if this is already documented.
the scripts I run are all part of a python package I’m working on, the packages are installed using conda with.

pip install -e

which makes a symbolic link from conda to my package’s directory.

When I run the experiment with guild it stores the source code of all the files in the package–which is good. However, when I try to restart the run, the program starts by using the main function in my
guild sourcecode folder–which is what I want-- but the moment it tries to use a module imported from its parent package it starts using code in the packages main directory instead of the version of that file saved by Guild. How could i get guild to use the files saved by guild?

2

Hello and welcome!

Packages are not copied per run but rather used from the environment (shared across runs). If you upgrade a package in the environment and then restart a run that uses that package, the new version is used.

From what I’m hearing, you want a separate environment per run. Guild happens to do this for remote runs, but not for local. I think this would be a good feature though as it provides a higher level of isolation and insulates restarts from any package changes.

As a workaround, you could run your operations over ssh+localhost using the --remote option. Something like this:

# ~/.guild/config.yml

remotes:
  localhost:
    type: ssh
    host: localhost

guild run your_op --remote localhost

Since your project is structured under a package, this should work without issue.

Running this way you’ll get a copy of the environment per run. You can see this by running guild open --path .guild — look under job-packages and you’ll see the required packages there. These are used for the environment when running the operation.

This is a bit of a pain — to run with the remote operation — but I think it’ll give you the isolation you’re looking for.

Let me know if this makes sense and how it goes if you try it.

3

Thanks Garrett. I will need to have a look at the documentation for the --remote option. although it sounds like it will solve my problem.

1 Like