I am trying to execute an experiment.py file that is called upon in guild.yml. The guild.yml file contains 5 flags for an svm experiment on the iris dataset. When I execute ‘guild run model:train’ on my local machine in a terminal, there are no issues and the flags are properly captured in the /.guild directory. However, when I execute the experiment in a Docker container, the flags are no longer being tracked by guild. Is there any way to debug this??
Or where in the source code are the flags picked up by guild to create the Flags file in /.guild directory.
Please help!
guild.yml file:
terminal output display of what in inside the /.guild directory: (notice flags is just an empty dictionary)
Guild is saving all created files (e.g. new run directory with code, tensorboard summaries, etc.) into the /root/.guild and some things into the /tmp (profiling files and some tensorboard stuff). Since you are running your guild code in docker because of this the guild is creating all of this in the container and at the end of the run when the docker is terminated all of these files are deleted. There is workaround there when you are calling the docker you have to add new volumes something like -v “folder_for_results_on_your_host:/root/.guild” -v “folder_for_tmp_results_on_your_host:/tmp”, this way you will have all runs out of the docker and they will persists.
For example I am using something like this:
docker run -it --rm --gpus device=none -v “$(pwd):/work_dir” -v “$(pwd)/guild_results:/root/.guild”-v “$(pwd)/guild_tmp:/tmp” docker_name:docker_tag guild run MODEL:OPERATION -y
work_dir is name of my working directory in docker
$(pwd)/guild_results: folder in my project folder where I want to save all guild runs results
$(pwd)/guild_tmp: folder in my project folder where I want to save all temporary guild results
MODEL: model name from your guild file (if you are using any)
OPERATION: operation name from your guild file (if you are using any)
I don’t know if you are running guild view in docker or not but in case you are running then you should also add the volumes into docker run command which you are using for the guild view. If you are running guild view on your machine (not in docker) then just change $(pwd)/guild_results with /root/.guild (and also for the temporary files if you are genereting any).
Thank you for your response!
I was able to resolve the issue.
The error persisted because in my guild.run() command, I only referenced the main() function from my experiment.py file as the argument for that run command–> guild.run(main), but I also needed to add a dictionary of flags I wanted to be tracked as a second argument. For reference, my final command looks like guild.run(main, flags_dict). This fixed the issue, and the flags from the guild.yml file are now properly being tracked and displayed in the /.guild output directory.
