Redirecting Standard Output

I have a guild file with the following lines:

train:
  description: Train a model
  exec: python -m model.utils.run > logfile

The train command works correctly, however it does not redirect the standard output to “logfile”. I am able to see the standard output in the “Log” tab of the guild view server, but this also contains the output of the tqdm progress bar used in the code, which is not ideal.

Using logger and tqdm with guild needs some tweaking becuase guild take over your logger steam handler.

In short you need to use a handler like this:

class TqdmLoggingHandler(logging.Handler):
    def __init__(self, level=logging.NOTSET, verbose=False):
        super().__init__(level)
        self.verbose = verbose

    def emit(self, record):
        try:
            msg = self.format(record)
            if self.verbose:
                tqdm.write(msg)
                self.flush()
        except (KeyboardInterrupt, SystemExit):
            raise
        except:
            self.handleError(record)


_stream_handler = TqdmLoggingHandler(verbose=verbose)
_stream_handler.setFormatter(formatter)
logger.addHandler(_steam_handler)

This should correctly direct your tqdm.write to the log file. The progress-bar on the other hand is a little bit tricky as basically you need to flush the progress bar to STDOUT but not to logger stream handler. See this for more details.
I would recommend simply use software like lnav to filter them out.

@teracamo thanks for the example!

@ap000 there are couple issues in play here. You would typically not redirect output to a file (e.g. logfile) because Guild does this by default. You can find the operation output (joined stdout and stderr streams) in .guild/output under each run directory.

You can view this file using:

guild cat --output [RUN]

However, as you’ve pointed out, Guild has a known issue when run with tqdm .

The workaround is to disable Guild’s output capture by setting the env variable NO_RUN_OUTPUT=1. In this case you can capture output yourself using file redirection.

You want something like this:

test-workaround:
  exec: bash -c "python -m model.utils.run 2>&1 | tee logfile"
  env:
    NO_RUN_OUTPUT: 1

This turns off Guild’s run output capture and directs your script output to logfile. The 2>&1 is optional, and includes stderr output in the result.

However! For this to work, you need to install 0.7.5.dev1, which has a fix to support this.

For anyone interested in more details (e.g. to work on this issues hint hint :wink: ) refer to the corresponding issue resolution doc.

1 Like

Thank you both for your responses! I was able to work around the tqdm issue with the NO_RUN_OUTPUT flag as an immediate solution and will look into using the provided TqdmLoggingHandler in the future. Thanks so much!

1 Like