Guild is saving all created files (e.g. new run directory with code, tensorboard summaries, etc.) into the /root/.guild and some things into the /tmp (profiling files and some tensorboard stuff). Since you are running your guild code in docker because of this the guild is creating all of this in the container and at the end of the run when the docker is terminated all of these files are deleted. There is workaround there when you are calling the docker you have to add new volumes something like -v “folder_for_results_on_your_host:/root/.guild” -v “folder_for_tmp_results_on_your_host:/tmp”, this way you will have all runs out of the docker and they will persists.
For example I am using something like this:
docker run -it --rm --gpus device=none -v “$(pwd):/work_dir” -v “$(pwd)/guild_results:/root/.guild” -v “$(pwd)/guild_tmp:/tmp” docker_name:docker_tag guild run MODEL:OPERATION -y
- work_dir is name of my working directory in docker
- $(pwd)/guild_results: folder in my project folder where I want to save all guild runs results
- $(pwd)/guild_tmp: folder in my project folder where I want to save all temporary guild results
- MODEL: model name from your guild file (if you are using any)
- OPERATION: operation name from your guild file (if you are using any)
I don’t know if you are running guild view in docker or not but in case you are running then you should also add the volumes into docker run command which you are using for the guild view. If you are running guild view on your machine (not in docker) then just change $(pwd)/guild_results with /root/.guild (and also for the temporary files if you are genereting any).