Operations as dependencies during checks

Troubleshooting
1

I have a guild model that looks something like this:

- include: source_code_config.yml
- model: _check
  extends:
    - source_code_config
  operations:
    _test_segmentation_training:
      steps:
        - run: segmentation:train dryrun=yes num_epochs=1 input_database="x.csv"
          expect:
            - file: experiments/best_model.pt
    _test_segmentation_testing:
      steps:
        - run:segmentation:test input_database="x.csv"
          expect:
            - output: "Testing done."
    _all:
      steps:
        - _test_segmentation_training
        - _test_segmentation_testing

I use this for integration testing my training and tests scripts.

The segmentation model looks something like this:

- model: segmentation
  extends:
    - source_code_config
  operations:
    train:
      main: scripts/training/train_segmentation
      flags:
        $include:
          - segmentation_flags
          - train_flags
          - common_flags
      requires:
        - prepared_data
    test:
      main: scripts/training/test_segmentation
      flags:
        batch_size: 1
        $include:
          - test_flags
          - common_flags
      requires:
        - prepared_data
        - trained_model
  resources:
    trained_model:
      sources:
        - operation: train
          select:
            - experiments
            - .guild/attrs/flags
          target-type: copy
          rename:
            - flags training-flags.yml # See https://github.com/guildai/guildai/blob/0.7.2/examples/upstream-flags/guild.yml

Now when I run

guild run segmentation:train

Followed by:

guild run segmentation:test

guild will automatically resolve the trained_model dependency and find the latest segmentation:train run.

If I instead run:

guild _check:_all

The _test_segmentation_training runs successfully, but the _test_segmentation_testing operation is not able to resolve the trained_model resource.

Ideally I would be able to run this pipeline as a step in my integration testing.

EDIT:

Using guild version 0.7.3

2

You’re running into a purported safeguard where steps can only see runs generated within the parent steps operation. To tell Guild to look at all runs, set isolate-runs: no within the step.

Here’s an example:

This is annoyingly subtle behavior—Guild should at least print a message pointing you to an answer.

Be sure to disable isolate-runs the two applicable steps/pipeline operations.

1 Like