Custom post-hoc analysis

I would like to do some custom post-hoc analysis on various things I’ve logged into tensorboard. I’m doing bootstrapping and want to calculate mean and confidence interval. I know there’s the ipy widget, and I’ve gotten semi far with that as it gives you avg_value etc. But I’d like to get access to all the values logged for, say, precision so that I can calculate the confidence intervals. I know this data is logged in tensorboard and tensorflow has some guides but they do not work with the version of tensorboard that comes with guild. How do you advise I would do something like this?

I can log this separately, sure but it ends up being a pain and cluttering up all my logs. I cannot visualize them the way I’d like in tensorboard, so I’d rather do this analysis ad-hoc when comparing runs myself.

This is not currently supported, but it’s a very reasonable request so I added it! It’s in master now and will land in the next release.

In the meantime, you can use this code to get the low level scalar detail:

from guild import ipy
from guild import tfevent

run = ipy.runs().iloc[0].run.value
for path, _run_id, scalars in tfevent.scalar_readers(run.dir):
    for tag, val, step in scalars:
        print(run.dir, path, tag, val, step)

API that you can use today

Note that this API is unofficial and could change (though that’s not likely). You’ll want to use guild.ipy when that feature is available. That will look like this:

from guild import ipy

scalars = ipy.runs().iloc[0].scalars_detail()
print(scalars)

0.7.1 API — not yet released

1 Like