Feature request: It would be great to use some other visualization tool such as wandb, comet, etc. We could for example specify some binding in the guild.yml file:
This way when Guild logs a metric, it would as well pass that metric onto the function specified in the guild.yml file. This also allows for any custom data processing. To make it easier, the user can handle importing the proper library and initializing experiment in their code. Using only guild to manage metric tracking makes is much easier and makes the code more portable (such as another user not wanting to use comet.ml).
While Guild supports output scalars, that’s really more for folks who don’t want to bother with a specific log library and would rather just scrape/parse their script output. If you’re going to far as to create a callback function for Guild, I would just log whatever you want directly and disable Guild’s output scalar support altogether.
Guild uses native TF event files for its scalar stores. You can log scalars using any number of libraries (e.g. tensorboardX, etc.) Along with that, you can log to any of the other experiment frameworks directly. If I’m understanding your feature suggestion, you’re doing that anyway by way of the callback function. By logging directly, you avoid the complexity of Guild scraping/parsing output and then also calling back into your codebase.
Yeah I guess it could be possible to just process the TF event files. Where are those stored? I did not see it in the docs (probably missed it somewhere). I guess the advantage of Guild is that it already has the infrastructure to capture the printed metrics. I like that as I want to avoid adding logging code everywhere in my files so I thought of using Guild for that.
Guild stores any TF event files under the .guild directory in a run. If you write your own, they’ll go wherever you write them, relative to the run directory root.
I see your point — it’s convenient to have Guild handle the screen scraping, which is already there and working. Guild could then be used as a means to get any experiment logged into various systems.
What specific use case do you want to achieve here? Do you want Guild experiments to show up in say MLflow?
Another approach, which has precedence, is to support an export to a format that one of these other experiment logging tools supports. E.g. you run your Guild experiments (which log to TF Event files) and then run something like guild export --target mlflow and Guild does the right thing to get your runs showing up in the target.
This could be more thorough as these other systems log other things besides scalars/metrics (e.g. flags, dependencies, generated files, etc.)
This would be great. The reality of collaborating in an ML lab is that each person has their preferred logging tool. A main reason why I use Guild is for the ease of metric logging without any logging code (logging setup simply requires a few lines to the guild.yml file, which is incredibly useful). Anyone can run the code without having to register for some service like comet.ml. Use case guild export --target mlflow (and other services comparable to mlflow) would be great.
Alternatively (or in addition), a very bare bones implementation to the idea in my initial post is:
Here you could simply forward all captured data to the process function in the logging.py file. This would allow for custom logging but also re-use the Guild scraping, saving a ton of time.
It makes me nervous to push Guild into log routing territory, especially with screen scraping, but I admit you make a strong point. It’s quite convenient to use Guild in this way — especially in a collaborative setting where you don’t want to start modifying other users’ scripts just to accommodate experiment tracking. After all that’s Guild’s secret formula
@Gilded would you mind creating a GitHub issue for this so we can track it as a feature request? Simply adding a link to this topic is enough for the issue body.
