`guild export` and underlying shututil dead-slow for samba drives

Troubleshooting
1

This is a very specific issue, but I wanted to raise it either way.

I am using guild export to export runs to a Samba share mounted in Linux. This is incredibly slow. I see here that guild uses shututil. This is a known issue for shututil and they suggest a monkey patch in this issue here.

I think I am going to experiment with ways to patch guild. Since we are using Azure, I want to see if I can utilize azcopy, which is orders of magnitude faster.

2

Thank you for this! If you have some success with this, I encourage you to submit a PR or a gist. I can help integrate and confirm with you that the fix/patch works in the next release.

3

I decided to abandon the Samba drive. It simply to slow and too much of pain to get to work properly in Azure.

If we used Amazon, we could utilize the guild push command to an s3 bucket. Instead I am now opting for storing guild runs in dvc, which has support for Azure.

4

I wonder if Guild could support DVC as a remote type. E.g.

# ~/.guild/config.yml

remotes:
  dvc:
    type: dvc
    # Config here as needed.

guild push dvc  # syncs runs with DVC

Do you think this would be useful, or are you thinking just place the $GUILD_HOME/runs under DVC control independently?

5

I am doing the latter now, i.e. placing guild runs under DVC control. That works OK for now, but more integrated support for dvc by guild would be amazing.