Resolving a nested ini/cfg file changes its format

Troubleshooting
1

First, thanks for the awesome tool!

I am having some trouble with the following behaviour. I use .ini config files with nested sections, and import the flags from there, but when Guild resolves the files and write a copy to the run dir, the format changes and the nesting is lost. I would need it to keep the same formatting.

I have a config file with nested sections, like:

[ingredients]
greens = cabbage

[ingredients.meat]
type = pork belly
marbled = true

[ingredients.carbs]
type = noodles

[ingredients.carbs.make]

[ingredients.carbs.make.water]
add_miso = true
container = cooking pot
temp = 100

And a guild file:

cook:
  exec: cat source_config.cfg
  flags-import: all
  flags-dest: config:source_config.cfg

Running this operation gives prints out the resulting config file in the run dir:

[ingredients]
carbs = {'make': {'water': {'add_miso': True, 'container': 'cooking pot', 'temp': 100}}, 'type': 'noodles'}
greens = paksoi
meat = {'marbled': True, 'type': 'pork belly'}

I can understand what happened, but for my purposes, this is a destructive change: I can’t read this config file inside the run.

Practically speaking, I use guild to run machine learning experiments backed by Spacy. I read in the created config file with confection (Spacy’s config handler), which doesn’t support reading those dicts. The upshot is that I can’t use guild to run sweeps. Would love to see a solution to this.

2

For your consideration, here is a snippet sketching a function with approximately the behavior I’m looking for. Modified from guildai/util.py at main · guildai/guildai · GitHub

def encode_cfg(data):
    import io
    import configparser

    cfg = configparser.ConfigParser()

    def encode_section(trace: list[str], d: dict):
        section_key = ".".join(trace) if trace else configparser.DEFAULTSECT
        if trace:
            cfg.add_section(section_key)

        # First treat the non-dict options in `d`.
        # Collect the dict options as we go.
        dict_options = []
        for k, v in sorted(d.items()):
            if isinstance(v, dict):
                dict_options.append((k, v))
            else:
                cfg.set(section_key, k, str(v))

        # Then go recursively into the dict options.
        for k, v in dict_options:
            local_trace = trace + [k]
            encode_section(local_trace, v)

    encode_section([], data)

    io = io.StringIO()
    cfg.write(io)
    return io.getvalue()


test_data = {
    "type": "recipe",
    "ingredients": {
        "carbs": {
            "make": {
                "water": {
                    "add_miso": True,
                    "container": "cooking pot",
                    "temp": 100,
                }
            },
            "type": "noodles",
        },
        "greens": "cabbage",
        "meat": {"marbled": True, "type": "pork belly"},
    },
}

print(encode_cfg(test_data))

Output:

[DEFAULT]
type = recipe

[ingredients]
greens = cabbage

[ingredients.carbs]
type = noodles

[ingredients.carbs.make]

[ingredients.carbs.make.water]
add_miso = True
container = cooking pot
temp = 100

[ingredients.meat]
marbled = True
type = pork belly
3

Hi @Camiel welcome and thank you for the report!

Yeah, I can see how this behavior is annoying - the radical change in format it is a bit surprising but I’m guessing this is the behavior in the Python INI support (perhaps later versions).

I’ll create a project that replicates and see what we can do to address it. Stay tuned!

1 Like
4

I opened an issue in GitHub to track this. We’ll look to integrate that in the followup release to 0.9.0 — it looks like the right approach! Thanks a lot for that :+1:

5

My pleasure, looking forward!

6

You can find the fix here Guild not preserving nested INI sections as expected · Issue #482 · guildai/guildai · GitHub, it’s merged into main, and hasn’t been released, but you can run from source code (via Install Guild AI under the “From Source Code” section). Let us know if that works for you.

2 Likes