Combine argparse and config files

Is it possible to combine flags from the argparse interface and config files?

Consider this pseudo example:

Project structure:

guild_combine_argparse_config_files/
├── guild.yml
├── model_flags.yml
└── train.py
# train.py
import argparse

class Model:
    def __init__(self, x, y):
        self.x = x
        self.y = y

    @staticmethod
    def from_config_file(cfg: Dict):
        return Model(x=cfg["x"], y=cfg["y"])


def main():
    pa = argparse.ArgumentParser("Main arguments for choosing higher order components.")
    pa.add_argument("--csv_path")

    config_file = load_config("model_flags.yml")
    mdl = Model.from_config_file(config_file)

    dataframe = load_dataframe(args.csv_path)
    train_model_on_dataframe(mdl, dataframe)


if __name__ == "__main__":
    main()

# model_flags.yml
- config: model_flags
  description: Collection of parameters for my model.
  flags:
    x:
      default: 5
      type: int
    y:
      default: 10
      type: int

# guild.yml
example:
  main: train
  flags:
    csv_path: "my_path.csv"

Can I get guild to recognize the flags specified in model_flags.yml? It would be cool if I could do something like:

guild run example x=2 y=10
guild run example # Just use default values in model_flags.yml

The way I do this now is by specifying x and y through argparse and then use $include operator in my guild.yml file for the model_flags.yml flags, but I still have to specify x and y through argparse.

I was wondering if you could do something like:

# guild.yml
example:
  main: train
  flags-dest: 
    - config:model_flags.yml
    - config:model2_flags.yml
  flags:
    csv_path: "my_path.csv"

I was looking through the hydra examples, but I don’t necessarily want to use hydra for this. I am fine with simple yaml config files or guild files.

Hi,

How about use the feature of batch files Runs, you can even combine

  1. argparse default value
  2. flag value in the guild.yml file
  3. batch files and batch from command line

note that the overwrite order is 3>2>1.

This is definitely something that we want. There are cases where you want to separate types of params because they don’t mix well — e.g. system related vs. model related or meta config vs op-specific config, etc.

I think your proposal is a good one. It is a bit more complicated because we really need a per-flag args-dest attr, so you can direct any given flag to the appropriate interface.

Short of having this feature, I would consider using a wrapper that used a single interface (e.g. argparse or globals) that generates the required YAML/config files + calls the wrapped script with the appropriate flag values. This is the classic “use Python to hack it!” approach, which you can often fall back on.