WARNING: Guild took 9.56 seconds to prepare runs. To reduce startup time, try running with ‘–skip-images’ or ‘–skip-hparams’ options or reduce the number of runs with filters. Try ‘guild tensorboard --help’ for filter options.
Adding skip-images or skip-hparams does not help.
I suspect this happens because I have a resource (symbolic link) to a directory with a lot of files. Is it possible to configure guild tensorboard to ignore this directory?
I found the -O option in: Command: tensorboard
but adding -O logdir=tb did not work (had to abort because it never finished).
There’s no easy workaround that I can think of. To me this behavior is arguably a bug. Guild is following symlinks to find run files that might be used for various TensorBoard plugins (e.g. projections, etc.) and I don’t understand why.
I’ll spend some time investigating and look at changing this this. There may be a good reason for the current behavior.
Either way I’ll address this in master. 0.7 is frozen - rc11 is the last release candidate barring some world ending bug that’s discovered.
Hello
I see, thanks for checking! I don’t think I had any trouble running from source when I did so previously.
I confirmed that the issue was the symbolic link. When I removed it, guild started tb very quickly.
I think this issue would be solved for my specific case if I could specify where to look for tb-logs in the guild.yml. I could maybe use -O logdir=dir/to/guild/runs/id/tb already but that’s quite a hassle
-O is meant to pass options along to TensorBoard without any knowledge of what’s being passed. I think logdir needs to be explicitly ignored (with warning message) as Guild takes over the function of setting up and specifying the logdir that TensorBoard sees.
I see what you’re getting at. I agree this should be in the Guild file. I hate to complicate things, but given the flexibility of the tool, where the script can put files anywhere it wants, I think the spec would have to follow the line of sourcecode and use Guild’s file select spec.
I think maybe this interface?
op:
tensorboard:
logdir: relpath-to-tb-files
In this case logdir is a fully supported file select spec like that used for sourcecode.
If you grab the latest from master that issue should be resolved for you. Just make sure you’re using Guild from source.You can run guild check and look for guild_install_location to confirm it’s running from the repo and not from an installed package. (You may need to run hash-r to clear your shell’s cached location of the guild exe.)
