One of the greatest features in
guild is the ability to so easily compare different runs using all the great visualization tools. This works great for comparing run metrics.
I work in robotics and computer vision and I usually need more advanced custom visualizations to describe performance of a run.
For an example, in robotics pose estimation you want a 3D line plot comparing the estimated pose trajectory compared to the ground truth trajectory. Currently in
guild you can easily compare the RMSE between runs, but not this example plot.
I have a couple of ideas to implement this myself, but I wanted to put it out there to see if other had some good ideas.
My first idea is to use the
guild.ipy Python interface in a stand-alone
jupyter notebook (i.e. indepedent of
guilds notebook feature). The notebook will be parameterized to take as input any number of
guild run-id and then create the comparison plots.
Another idea is again to use
guild.ipy together with
streamlit to create a more interactive experience.
These ideas are unfortunately quite separeted from the rest of the
guild environment. I was wondering if there was a way to add a plugin to the
guild view command. In this way the user could write up his own comparison tools and that could be integrated into
guild view just as
view in TensorBoard currently is. For an example, maybe
guild could support a notebook-plugin. So the notebook will be executed per run and the associated HTML notebook could be displayed in
guild view. Then you would have multiple HTML notebook for each run and could compare them in
Open for any input and welcoming discussion!