Anyone using Jupyter Notebooks?

I realize this is a Guild AI community and so the answer to this might very well be 100% NO :slight_smile:

Jupyter Notebooks are extremely popular. That means that a lot of data science work is conducted in Notebooks. Unfortunately, the ad hoc workflow that make Notebooks so popular doesn’t lend itself to strictly controlled experiments.

I think Notebooks actually are pretty good for experiments, if used correctly. Unfortunately, it’s quite hard to use them correctly.

To properly represent an experiment, a Notebook:

  • Must be copied to a new, pristine file
  • If flags are changed, must be modified to reflect the experiment-specific flag values
  • Run from top to bottom by a script, not a human
  • Once run, must not be re-run

The copied-and-run Notebook can then be used as a reliable experiment artifact.

0.7.1 has preliminary support for this. It lets you run a ipynb file directly.

guild run my-notebook.ipynb

You can also use a new notebook operation attribute in a Guild file:

# guild.yml

op:
  notebook: my-notebook.ipynb

If you’re using Notebooks — or working with folks who are — I’d love get your input on this feature.

1 Like