Cross Validation

General
1

Hello everyone,

I just started using guild.ai and have a question regarding cross-validation during hyperparameter optimisation.

As I understand it, guild runs my python scripts/jupyter notebooks externally and can change variables (hyperparameters) and read from the output. I created a small example using scikit-learn and skorch where I construct a model and dataset and then train the model.

So far, everything has worked fine. I can see all runs with their results and hyperparameters. However, I started wondering how I could integrate cross-validation with this. I saw some remarks regarding “cv” in the guild source code but couldn’t find any examples in the documentation.

Does guild intercept the training data and manipulates it into several smaller datasets? Or do I have to change my code to integrate cross-validation?

I would be happy if you could shed some light on this!

Thank you
Jan-Lucas

2

To add some thoughts to this:

If cross-validation is handled explicitly in the training script, multiple results (runs) can be created in the same way as the grid search produces multiple results (runs). E. g. training a CNN three times during cross-validation creates three different loss curves.

The advantage of handling cross-validation with guild (if it’s not already possible) is that each run is really one single run.

3

Hi @jlsachse! This is a great question and gets to an upcoming feature “summary operations”.
We wrote up a sample project that shows how you’d be able to implement CV using Guild and includes separate data prep, training, and summary operations.

This demo is waiting on the summary operations feature, however.

For the time being I recommend that you implement cross validation as a single operation. This would follow the standard methods outlines in this sklean tutorial.

Because cross validation tends to iterate over all possible folds (given some number of divisions) it is sensible I think to implement this as a single training run. I offhand can’t think of a reason to use Guild to drive separate training runs, other than reflecting the summary scores separately. But this is not really what cross validation is useful for, rather it’s the average and std deviations of the scores when run across all of the folds.

Is there a reason you’d like to use Guild for running each fold separately, other than reflecting each pass as a separate run?

4

Hi @garrett,

Thank you for the quick answer and example. To set things into context, here are some details about my project:

I’m working with a small dataset, and I want to optimise multiple models with some hyperparameter search. I’ve not yet decided on the algorithm, but it will likely go with Bayes. Since the dataset is small, I think it makes sense to perform nested cross-validation.

I have two options:

  1. Integrate the nested cross-validation with guild ai hyperparameter optimisation + scikit-learn. This would be great for logging but also make the nested cross-validation seem more complicated as the outer and inner loops would be switched. I must also tightly control the random seeds to keep the splits constant across runs.

  2. I don’t use guild ai hyperparameter optimisation and use sickit-learn entirely for optimisation. This would hinder logging a bit, though.

For my first question: I only wanted a way to log loss functions in tensorboard across cross-validation runs. I use skorch to make my PyTorch model compatible with scikit-learn, and they offer a tensorboard logger. However, the logger doesn’t work with cross-validation as scikit-learn copies the model before runs, and there is a problem with the summary writer where it can’t be copied… So I tried to work around this with guild ai.

It would be great to hear your thoughts on this.