Where does the code runs from?

If I understood correctly, the source code is copied to the run unique directory upon guild run execution, and locally created files/folders are created in this directory.

My question is where does the code run? Is the copied version of the code executed, or the original?
I have a line in the code that creates a variable with the code file absolute location:

Path(os.path.dirname(os.path.abspath(__file__))).parent

which returns the original code path under guild run. I would expect this to point to the file in the copied version of the code.

Is this behavior expected? If so, can you please explain the logic behind this?

Regards,

Excellent question. The code is run from the source code target under the run directory (by default .guild/sourcecode) and NOT from the project.

This very much is by design. We originally ran the project code from the project location. This is dangerous because there are ongoing race-conditions as the operation runs. Any changes to the project code can be unintentionally picked up by the operation process depending on when that code is loaded and when the change is made.

This problem is particularly acute with staged runs. The workflow for staged runs is to stage something so that you can continue to work locally. You’re assured to break runs, badly, if you run from the project location.

The code sample that you use should resolve to a location under the run directory — not under the project location. If that’s not the case, that’s big problem and we need to look into it.

I assume this variable is later used to find other files. There are a couple approaches you can take here:

  • Include those files as source code. See this link for more details.
  • Configure your program with a variable or command line option that points to a location for these files and setup the operation to resolve those as file dependencies as needed.

It’s quite common to reference files that sit alongside source code. In this case, I’d argue that those files are part of the source code. They’re not really file dependencies.

If the files are not sitting alongside source code files — e.g. you’re reaching outside the source code root (e.g. $source_code_path/../../data/...) then I’m inclined to view these as required files. Your program could expose a variable to point to their location. You’d use the current scheme as the default, but then allow a caller to specify a different location. Then in Guild you setup your run directory as needed and then set that location to the run directory path.

Thank you for the detailed answer. I was hoping that the code is run from the source code target under the run directory, exactly for the reasons you have mentioned – so I am glad you confirmed this.

I have tried to create a minimal working example to show the issue, but it was not re-created in that small example. I do see this buggy behaviour in my project though - I will see if I can create such a minimal working example.

1 Like

OK, I have realized what went wrong. I have added the project’s main directory to the PYTHONPATH environment variable for easier import, and though the source code file holding that path variable was copied as part of the source directory, the code was imported from the original path instead of the copied version.

I would not say it is a bug of the guild - but it is a delicate issue one should be aware of. Using non-relative imports is probably bad practice, and it is especially problematic when using tools such as guild that copy the source code directory to a different folder, resulting in unexpected results.

In order to recreate, setup the following:

  1. Under some directory in the file system (I will assume here it is the /home/tmp/src/ directory) have the following files (with their content):

    • config.py
    import os
    my_path = os.path.dirname(os.path.abspath(__file__))
    
    • test.py
    import src.config as cfg
    print(cfg.my_path)
    
  2. Set PYTHONPATH environment variable as follows:

    export PYTHONPATH=:/home/tmp
    
  3. Run from within the /home/tmp/src/ directory

    python test.py
    

    and you’ll get the expected output: /home/tmp/src/

  4. Run from within the /home/tmp/src/ directory

    guild run test.py
    

    and you’ll also get the output: /home/tmp/src/ (although the config.py file is copied to the unique run directory, it will not use it (obviously).

  5. If you change the import command in test.py to

    import config as cfg
    

    you’ll get the expected output of the unique guild sourcecode folder when running under guild.

Thanks for the write up!

What you’re seeing there is by design. You’re treating src as a package, which must somehow be available when your script runs. You’re making it available by including the package location in PYTHONPATH. When run, your script will load from that location.

To track down what’s going on, you’d take these two steps:

  1. Show the source code layout for the run.
guild ls --sourcecode
  1. Show the PYTHONPATH used for the run.
guild runs info --env | grep PYTHONPATH

If you’re running Windows, omit the grep filter and just lookup the value for PYTHONPATH.

This information unambiguously answers the question, “what’s being run?”

To configure this scenario properly, you need a Guild file. In this case, I’ll locate it in the parent directory of src since you have src as a module (this is anti-pattern - see note below).

test:
  main: src.test

This is the only way to run a script in a package. Guild doesn’t support running modules directly.

When you run step 1 step above, you see that Guild copies the full src package hierarchy as source code. This makes the src package available under the first location in PYTHONPATH, which you can see by running step 2 above.

1 Like

I think the main point here, for the described case, is that you either need to use relative imports to the file location, or to the directory you are running from (and for the later, make sure guild runs from that intended directory).

Anyway, I don’t think it’s a guild issue that needs a resolution, just something to be aware of.

Thanks

There are two independent points reflected in this thread and I want to make this clear for others. Your conclusion is incorrect. You don’t need to use relative imports. In fact, you’re not even using relative imports (I elaborate below).

The fundamental problem is that your original code relies on a src package and that package is not available in your run source code directory. It’s as simple as that.

Guild runs your original code without modification perfectly well. I’ve provided a two line Guild file that you can use. I’ve provided a two step method for confirming that the source code layout and Python path is as expected. With this you can tackle this sort of problem without head scratching.

What you’re describing is the ant-pattern implicit relative imports. This is no longer a relative import in Python 3.

A relative looks like this:

from . import config as cfg

Unless the importing module is in a package, the import fails with a message along the line ValueError: Attempted relative import in non-package.

You’re recommending the use of absolute imports while incorrectly thinking you’re using relative imports. This is bad advice. This discussion about PEP 328 highlights the problems that Python 3 solves regarding absolute and relative imports.

Guild handles this correctly. You need to understand the tool and use it as designed. My previous post shows you how.

Thanks for clearing things up. I guess I used the wrong term of relative import here, and I think I did mix up a few things in my previous reply above (thanks for the PEP 328 link), but now I am not sure this is completely clear to me.

What I meant originally was, that since python adds the parent of the running script to the system path (at location 0), you can use import config as cfg and it will find the correct module whether you run it using python directly, or using guild run - either from the /home/tmp/src directory

  • python test.py
  • guild run test.py

or from the /home/tmp directory

  • python src/test.py
  • guild run src/test.py

Actually, adding the suggested guild file, and running guild run test, does not work anymore in this case as the src directory is no longer added to the system path (it only works correctly if I use import src.config as cfg).

So I guess my questions are:

  1. Why does the src directory not added to the path when running test with your suggested Guild file?
  2. What is the recommended way to use guild when we are using a more complex directory structure (and wish to run a snapshot of that entire structure)? For example:
    project/                      # root folder of the source code project
       packA/               
          main_a.py
          subA/              
             a.py
       packB/
          b.py
    
    where main_a.py is the main script to be run, that needs to import both from a (which is in subA) and from b (which is in packB), and we wish Guild to create a snapshot of the entire project source code folder (and obviously run from the snapshot and not from the original location).

Also, I found this issue resolution, but haven’t had a chance to look in the details: https://github.com/guildai/issue-resolution/tree/master/195-support-snapshotting-of-non-project-libraries

Hi @garrett!

There is some confusion.

In Get Started is written:

When Guild starts an operation, it executes these steps:

  1. Create a new run directory in the runs subdirectory of Guild home
  2. Initialize the run directory with run metadata in RUN_DIR/.guild
  3. Copy operation source code to RUN_DIR/.guild/sourcecode
  4. Resolve dependencies (you don’t have any yet — you learn about this in File Dependencies below)
  5. Run the main module using RUN_DIR as the current directory

In a post above you write

The code is run from the source code target under the run directory (by default .guild/sourcecode )

that working directory is RUN_DIR/.guild/sourcecode. I really wish that was the case, but at least in version 0.7.0.post1 it is RUN_DIR. And this behavior breaks some bash scripts that assume that they are running inside sourcecode (especially when one script is calling the other). Can you check this up, please?

Guild saves source code to RUN_DIR/.guild/sourcecode in 0.7.0.post1. This location will likely never change – and if it does there will be ample notice with deprecation warnings. We will never intentionally change an interface like this without warning.

At least by design, the only way Guild saves source code to another location is if dest is set for the operation sourcecode config. If you’re seeing Guild save source code directly under the run directory without this config, we need to investigate.

To test this in a simple case, could you create an empty file test.py in a new directory and run it?

mkdir /tmp/guild-test && \
touch /tmp/guild-test/test.py &&  \
guild run some_dir/test.py -y

(If you’re running a non-POSIX shell this command doesn’t work. Just use whatever steps you’re comfortable with.)

Then view the list of files for the run:

guild ls --all

You should see test.py under .guild/sourcecode. If you don’t, please let me know what guild check shows (you can omit any system specific paths - I only need the Python and OS versions).

If you do see test.py under .guild/sourcecode, we need to look more closely at your operation to see what’s going on there.

I’m sorry that I wasn’t clear enough. My confusion was about 5th point in the quoted text.

The source code is being copied into RUN_DIR/.guild/sourcecode, no problem with it. But the working directory (the directory the code starts running from) is RUN_DIR and not RUN_DIR/.guild/sourcecode.
I’ll explain you why it is important to me. I work with Kaldi framework for ASR. It has really complicated directory structure, but it can be simplified into following: it has running script run.sh and three subdirectories with a bunch of bash and python scripts: steps, local and utils. And the scripts assume being run from the directory where run.sh is located, and reference each other in that way.

Guild has always run operations from RUN_DIR — i.e. the current directory is the run directory. It has never run from RUN_DIR/.guild/sourcecode. This will not change — it’s a central design of Guild.

The Python system path is configured to contain the source code directory (default is .guild/sourcecode). The directory may contain subdirectories if they are included in the main spec (e.g. main: some-subdir/my_module).

Are you running run.sh from Guild or from a Python module (e.g. using subprocess)?

You can run the script directly with Guild using the exec operation attribute. In this case you’d need to specify it this way:

op:
  exec: .guild/sourcecode/run.sh

If you want to copy this code into RUN_DIR, configure your operation this way:

op:
  exec: run.sh
  sourcecode:
    dest: .

Thank you for the clarification and examples.

  1. Currently I’m running run.sh from Guild using exec.
  2. I did not consider possibility dest could be relative path. If I set dest: ., would I retain the ability to compare source code between different experiments and other source code related operations, like guild ls --sourcecode?
  3. Suppose I would like to keep the source code scripts in sourcecode. If I create auxilary script
cd .guild/sourcecode
run.sh

and run it through Guild, it should mitigate the problem, right?

  1. One more question: do Guild have some variable to reference project directory (where guild.yml is located) ?

You should be able to compare source code when you change the destination, but I see there’s a bug in 0.7.0.post1. Good catch!

You can just do this in the Guild file:

op:
  exec: bash -c "cd .guild/sourcecode && ./run.sh"

@garrett Just bumping it up as I think it got lost in the thread… Do you think you can answer the two questions I have asked in my previous post?

I spot one question that I missed. You can get the directory where guild.yml is located via the PROJECT_DIR environment variable. If there’s something else that’s not clear, please re-ask here so I can better know what I’m missing.

Sorry for not being more clear. I was referring to the questions posted by me in this thread on Aug 12th.

Would you please restate the questions here? You linked back to this topic, which I’ve reread and believe I have answered.

Sure. The two questions are not really related to each other, but they are both relate to the original question and the provided answers.

  1. What is the recommended way to use guild when we are using a more complex directory structure (and wish to run a snapshot of that entire structure)? For example:

    project/                      # root folder of the source code project
       packA/               
          main_a.py
          subA/              
             a.py
       packB/
          b.py
    

    where main_a.py is the main script to be run, that needs to import both from a (which is in subA ) and from b (which is in packB ), and we wish Guild to create a snapshot of the entire project source code folder (and obviously run from the snapshot and not from the original location).

  2. Above I have given an example of some code and you have suggested a workaround with which I encountered an issue. Assume the following structure and code:

    /home/tmp/src/                      # folder of the source code project
       - config.py:
             ----------------------------------------------------
             import os
             my_path = os.path.dirname(os.path.abspath(__file__))
             ----------------------------------------------------
       - test.py:
             ----------------------------------------------------
             import config as cfg
             print(cfg.my_path)
             ----------------------------------------------------
    

    Now, since python adds the parent of the running script to the system path (at location 0) , it will find the correct module whether you run it using python directly, or using guild run - either from the /home/tmp/src directory, using

    • python test.py
    • guild run test.py

    or from the /home/tmp directory, using

    • python src/test.py
    • guild run src/test.py

    However, if I use your suggested solution of adding the following configuration Guild file

    test:
        main: src.test
    

    to the /home/tmp directory, and running from there guild test, the code fails to run - investigating this I found that the reason is that the src directory which is created under the unique run dir is not part of the python path, and the config module cannot be found. Is this a bug?

For question 1, assuming guild.yml is located in the root directory (recommended) you’d use this:

# guild.yml

op:
  main: packA.main_a

For question 2 you’d use this:

# guild.yml

test:
  main: src/test

When you specify src.test you specify a module. When you use src/test you specify first a path relative to the source root and then a module name (the part after the path).

1 Like