Hello, I noticed that when the label of a run has a trailing space, guild tensorboard cannot start.
Example
If I run my experiment like so,
guild run XGBClassifier:train --label "TEST"
then TensorBoard opens fine.
But when I put a space behind the label name,
guild run XGBClassifier:train --label "TEST "
then guild tensorboard results in this trace:
Preparing runs for TensorBoard
ERROR: error removing C:\Users\user01\AppData\Local\Temp\guild-tensorboard-13xxcmby: [WinError 2] The system cannot find the file specified: 'C:\\Users\\user01\\AppData\\Local\\Temp\\guild-tensorboard-13xxcmby\\570734cd XGBClassifier_train 2023-02-27 16_14_43 TEST '
Traceback (most recent call last):
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\lib\site-packages\guild\util.py", line 646, in _windows_symlink
subprocess.check_output(args, shell=True, stderr=subprocess.STDOUT)
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 424, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['mklink', 'C:\\Users\\user01\\AppData\\Local\\Temp\\guild-tensorboard-13xxcmby\\570734cd XGBClassifier_train 2023-02-27 16_14_43 TEST \\fold0\\mfa_loadings.png', 'C:\\Users\\user01\\AppData\\Local\\Programs\\Python\\Python39\\venv\\proj01\\.guild\\runs\\570734cd4c664dec9a57db2f12b9b3b2\\fold0\\mfa_loadings.png']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\Scripts\guild.exe\__main__.py", line 7, in <module>
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\lib\site-packages\guild\main_bootstrap.py", line 29, in main
_main()
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\lib\site-packages\guild\main_bootstrap.py", line 54, in _main
guild.main.main()
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\lib\site-packages\guild\main.py", line 30, in main
main_cmd.main(standalone_mode=False)
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\lib\site-packages\click\core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\lib\site-packages\click\core.py", line 1055, in main
rv = self.invoke(ctx)
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\lib\site-packages\click\core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\lib\site-packages\click\core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\lib\site-packages\click\core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\lib\site-packages\guild\click_util.py", line 235, in fn
return fn0(*(args + (Args(**kw),)))
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\lib\site-packages\guild\commands\tensorboard.py", line 130, in tensorboard
tensorboard_impl.main(args)
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\lib\site-packages\guild\commands\tensorboard_impl.py", line 41, in main
_run_tensorboard(args)
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\lib\site-packages\guild\commands\tensorboard_impl.py", line 83, in _run_tensorboard
monitor.run_once(exit_on_error=True)
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\lib\site-packages\guild\run_util.py", line 96, in run_once
self._refresh_logdir(runs)
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\lib\site-packages\guild\run_util.py", line 103, in _refresh_logdir
self.refresh_run_cb(run, run_path)
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\lib\site-packages\guild\tensorboard.py", line 279, in f
return _refresh_run(run, run_logdir, state)
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\lib\site-packages\guild\tensorboard.py", line 288, in _refresh_run
_sync_run_file_links(run, run_logdir)
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\lib\site-packages\guild\tensorboard.py", line 474, in _sync_run_file_links
_add_missing_run_file_links(run, run_logdir)
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\lib\site-packages\guild\tensorboard.py", line 480, in _add_missing_run_file_links
_ensure_run_file_link(src, link, link_dir)
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\lib\site-packages\guild\tensorboard.py", line 526, in _ensure_run_file_link
util.symlink(src, link)
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\lib\site-packages\guild\util.py", line 630, in symlink
_windows_symlink(target, link)
File "C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\lib\site-packages\guild\util.py", line 650, in _windows_symlink
raise OSError(e.returncode, err_msg) from e
PermissionError: [Errno 1] The system cannot find the path specified.
I think the space is preventing mklink from parsing the path arguments correctly.
Why it is problematic
When no label is passed to guild run, the experiment flags are automatically truncated to create a label. However, sometimes the flags just happen to be the correct length where the truncated label has a trailing space. Thus the error occurs.
Guild details
Here is the output of guild check
guild_version: 0.9.0
guild_install_location: C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\lib\site-packages\guild
guild_home: C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\.guild
guild_resource_cache: C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\.guild\cache\resources
installed_plugins: config_flags, cpu, dask, disk, dvc, exec_script, gpu, ipynb, keras, memory, perf, python_script, quarto_document, queue, r_script, resource_flags, skopt
python_version: 3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)]
python_exe: C:\Users\user01\AppData\Local\Programs\Python\Python39\venv\proj01\Scripts\python.exe
platform: Windows 10 AMD64
psutil_version: 5.9.4
tensorboard_version: 2.11.1
cuda_version: 11.7
nvidia_smi_version: 517.40
latest_guild_version: 0.9.0