User defined run identifiers

RFC
1

Summary

This RFC seeks input on the creation of a run identifier scheme that uses user-defined strings.

No part of this RFC proposes breaking changes.

This proposal is awaiting feedback

Problem

Guild uses run IDs in various applications:

  • Uniquely identify a run in a list (e.g. guild runs)
  • Identify a run when specifying an operation dependency (e.g. guild run test run=<run Id>)

These run IDs are not meaningful or intuitive to users and therefore hard to use.

Background

Guild uniquely identified each run using a globally unique identifier. These identifiers are represented as hexadecimal encodings of 128 bit UUIDs according to RFC 4122.

Guild use UUIDs to ensure uniqueness across both local and remote runs. Guild runs may be freely copied across systems without concern for ID collisions.

This proposal addresses the use of user-defined run identifiers across a number of applications:

  • How is this value specified
  • Can there be multiple such identifiers that point to a single run?
  • What are the implications of multiple runs having the same identifier?
  • How is this value displayed in a listing?
  • How is this value used to identify runs
    • List filtering
    • Command selection
    • Dependency selection
  • How is this value different from a run label, if it is?

Proposed Approach

This proposal uses Guild’s existing tag support to provide a sort of user-defined ID to runs. The use of tags in this case augments Guild’s use of run IDs and does not change any run ID related features or underlying implementations.

Changes to Guild (see below for details for each change):

  • Find a required run using run tags, in addition to run IDs
  • Option to tag command to apply tag to label
  • Option to tag command to apply auto-generated tag to label
  • Option to run command to apply auto-generated tag
  • Option to runs list command to show tags

Find runs using tags

Under this proposal, users can specify a run tag to identify an upstream run for an operation dependency.

Here’s a scenario:

  1. Generate a run
guild run prepare-data
  1. Later, through examination or via automation, determine that the run is of interest and tag it using guild tag --auto-label — this tells Guild to generate a human-readable tag and apply it to the run label
guild tag --auto-label

You are about to auto label the following runs:
  [003fc951]  prepare-data  2022-09-30 19:46:01  completed
Continue? (Y/n) 
The following run have been auto-labeled:
  [003fc951]  prepare-data -> redrobin
  1. Whenever the run-of-interest is needed for a downstream run it can be specified using it’s auto-generated label
guild run train data=redrobin

This commands tells Guild to select a run with a tag redrobin.

If there are multiple runs with tag redrobin Guild selects the most recent non-error run.

Tags would be matched on the whole word and be case sensitive.

One might argue that a tag match should be a substring (contains) match and be case-insensitive. However, this more flexible approach might be better served by a general purpose query spec, which could apply selection criteria across all run metadata, not just tags. See Generalized run selectors below for more detail.

Options to tag runs

This proposal entails a handful of enhancements to run commands to make tagging easier:

  • Options to the run and tag commands to apply an auto-generated tag to a run
  • Option to the tag command to simply adding a tag with label sync

When generating a new run, the --auto-tag option will generate and apply a human readable tag to a new run and include that tag value in the run label.

guild run prepare-data --auto-tag

You are about to run prepare-data (auto tag 'bluesparrow')
Continue (Y/n)

The tag command is enhanced with two options: --label <tag> and --auto-label. The --label option is syntactic sugar for --add <tag> --sync-label. --auto-label generates a human readable tag and applies it as per the --label option. This allows a user to more quickly apply a tag that appears in the label.

Show tags in runs list

This enhancement is not strictly needed for this proposal, but is included as a matter of completeness.

Guild supports the ability to show comments in runs list but does not support showing tags. That feature is included in this proposal to facilitate use of tags when referring to a run.

guild runs --tags

[1:df191e35]  train.py  2022-10-04 11:29:15  completed  [best, cifar, top-10] x=0.1
[2:302dd923]  train.py  2022-10-04 11:37:38  completed  [cifar] noise=0.1 x=0.2

In this case, Guild shows the list of tags in square brackets in the label column, preceding the label.

Alternative Approaches

Minimal change - use tags for run IDs

Guild currently supports tagging but does not support using tag values when referring to runs. So the command guild run prepare-data=redrobin will not work.

To support user-defined run IDs, we would need to implement run resolution using tags.

Drawbacks:

  • This limited approach may not provide a sufficient level of usability to address the underlying problem/request.

Use labels

We might consider supporting label matching to identify a run, rather than tag matching. This would motivate users to sync tags with labels, thereby ensuring that a tag appeared in a runs list.

Drawbacks:

  • Label matching is not as explicit as tag matching because tags are discrete strings whereas a tag-in-a-label is a substring.

  • It forces users to sync a label with tags when simply tagging a run would otherwise work.

Special tag type - e.g. “key tag”

The main proposal above relies on propagating a tag to the run label to make it visible in default listings. If a tag value is not synced with the run label, there’s no way to see the tag unless --tags is used in the runs list. As labels are used across several applications (e.g. in Guild View, Compare, and the VS Code extension), the approach of least impact is to leverage labels to show tags.

We might consider introducing a new run type: a key tag. A key tag is a special tag that can be used to identify a run. Key tags always show up in listings without the need to sync to a label.

Drawbacks:

  • This introduces yet-another-user-label concept, on top of labels and tags.

  • If runs were limited to a single key tag, it would less flexible than the proposed approach, which lets users select runs on any tag.

  • If runs could have multiple key tags, there’s little difference between this alternative and the proposed approach. The only difference is that key tags automatically appear in runs lists.

  • Runs lists would need to be modified to show key tags, in addition to labels. By using labels (the proposed approach) none of the current runs lists would need to change.

Generalized run selectors

This proposal is limited in its scope to user-defined run identifiers. It expands Guild’s support for run selection to include tags, as well as run IDs. One might wonder why Guild doesn’t extend this to support run selection for any run metadata. For example, can a user select a run with flags or scalars meeting certain criteria?

This might be a supported syntax:

guild run train data="?label contains dogs-and-cats and sample-size > 1000"

The application tags as “user defined” run IDs could be invoked using something like this:

guild run train data="?tagged redrobin"

Drawbacks:

  • The syntax ?tagged redrobin is too complicated to be used as an ID.

Note that this feature does not preclude the proposed approach.