How can I export all scalar values for all steps?

General
1

When I use guild compare I only see the scalar values for the last step.
I can see all the scalar values for each step of each run in the tensorboard interface - but how can I export this data?

1 Like
2

Hello and welcome!

Unfortunately no, but this is something Guild should do. I’ll make sure this gets into a near release as I think a lot of users could benefit from it!

In the meantime, here’s a snippet that will do this in Python.

from guild import var, tfevent

for run in var.runs():
    for _path, _digest, scalars in tfevent.scalar_readers(run.dir):
        for tag, value, step in scalars:
            print(tag, value, step)

It may not help in this case, but as background info you can define alternative columns for an operation to show different summaries of a scalar. As you point out, Guild shows last value but you can control this pretty easily. Check out Columns in Guild File Reference for details.

That functionality should be exposed in the Python API as well.

3

Would you mind terribly opening a GitHub issue for this? That way we can track resolution of this and discuss specifics as needed. That would be super helpful.

4

Thanks for the answer.
I have opened an issue here: https://github.com/guildai/guildai/issues/209#issue-656462230

2 Likes
5

0.7.1 (pending release) has a new feature that’s available in the ipy module.

>>> from guild import ipy
>>> runs = ipy.runs()
>>> runs.scalars_detail()
                                                 run    path   tag       val  step
0  <guild.run.Run '71d1768d53084e60bbd184da54f5c9...  .guild     y  2.000000     0
1  <guild.run.Run '0cc8862774f9429e849e78ec36d06c...  .guild  loss  0.534156     0
2  <guild.run.Run 'e98959d017754e628dea941501c757...  .guild  loss  0.730712     0
3  <guild.run.Run '2fe86de00bf3467384d73d8e694b6a...  .guild  loss  0.442781     0
4  <guild.run.Run '823d5678959d45338dcd63599d7f4d...  .guild  loss  0.520624     0
5  <guild.run.Run 'aeac83d788d24cddad530d42fbe218...  .guild  loss  0.598999     0

The scalars_detail() method returns a dataframe with the logged scalar values. This is similar to the code example above but officially integrated and easier to work with, assuming you’re okay using Pandas as the interface (Guild doesn’t require Pandas because it’s quite heavy and not every project will want to install it).

Implementing this into the CLI guild compare is trickier as Compare is geared for comparing runs at this higher level — i.e. showing a single summary values per run using a reduction (e.g. min, max, average, etc.) of multiple logged steps.

I see the value though of exporting the TFEvent scalars as CSV files for loading and analysis. I think this could land as a sub-command as the little known tensorflow command.

For example (this is hypothetical and NOT yet implemented or planned):

guild tensorflow export-scalars

But given the scalars_detail() method above, I wonder if this command is necessary.

What do you think?

1 Like