train:
description: Train a model
exec: python -m model.utils.run > logfile
The train command works correctly, however it does not redirect the standard output to “logfile”. I am able to see the standard output in the “Log” tab of the guild view server, but this also contains the output of the tqdm progress bar used in the code, which is not ideal.
This should correctly direct your tqdm.write to the log file. The progress-bar on the other hand is a little bit tricky as basically you need to flush the progress bar to STDOUT but not to logger stream handler. See this for more details.
I would recommend simply use software like lnav to filter them out.
@ap000 there are couple issues in play here. You would typically not redirect output to a file (e.g. logfile) because Guild does this by default. You can find the operation output (joined stdout and stderr streams) in .guild/output under each run directory.
You can view this file using:
guild cat --output [RUN]
However, as you’ve pointed out, Guild has a known issue when run with tqdm .
The workaround is to disable Guild’s output capture by setting the env variable NO_RUN_OUTPUT=1. In this case you can capture output yourself using file redirection.
Thank you both for your responses! I was able to work around the tqdm issue with the NO_RUN_OUTPUT flag as an immediate solution and will look into using the provided TqdmLoggingHandler in the future. Thanks so much!