Support for Azure Blob Storage as remote

General
1

I see that guild support S3 for remote storage of runs. Are there any plans on supporting Azure Blob Storage?

2

Yes, there’s a general plan to add remote support for both Google Cloud and Azure. This support will probably land after a handful of other critical features though, so it’s not on the short term road map.

This should not be hard though given guild.remoted.s3 as a pattern, which could be copied and modified as needed. It looks like Azure supports rsync style copies via https://docs.microsoft.com/en-us/azure/storage/common/storage-ref-azcopy-sync.

If you’re up for pursuing this I can help with any issues. I’d start this off by creating a branch and stubbing out the module to fill in with the applicable azcopy commands (or whatever client tool makes most sense).

3

Sounds good!

I don’t have much experience with Azure APIs myself, but I can try to give it a go. Would love to contribute to this project.

4

I started working on this and will let you know when I have something to try.

5

Okay, a first cut Azure blob storage implementation is in master. If you install Guild from source code you can test this out.

Create the remote along this line:

# ~/.guild/config.yml

remotes:
  azure-blob:
    type: azure-blob
    container: <url of container>

You need to install AzCopy. Run azcopy login and authenticate with a user who can read and write to the container. It’s a good idea to test this with some commands to azcopy first.

Note that the user must have the “Storage Blob Data Contributor” role. If the authenticated user does not have this role assignment for the specified container, you’ll get 403/auth errors.

With that configuration you should be able to push and pull runs to the remote Azure container as you can do with S3 or ssh based remotes.

There’s a few outstanding issues:

  • The metadata sync that Guild uses to get remote run metadata, which it uses to show runs, currently downloads all remote run info - not just metadata. The implementation uses azcopy sync which is unfortunately limited in its support for include/exclude patterns.

  • There’s no support for non-permanent run deletion. And therefore no support for purge and restore commands. This is due to some limitations with azcopy’s support for copy and move operations.

If you’d like to look more closely at this, check out guild/remotes/azure_blob.py in the source. The TODO items are marked in comments.

I think to ship this as a proper feature we’ll need to address these TODO items. This is a solid start but there’s work remaining.

6

Awesome. I’ll test it out during this week and let you know how it goes. Any reason you chose to use azcopy and not the Azure Python API?

7

Just a preference for the most mainstream method of interfacing. I looked over the Python SDK briefly to see if they supported a sync/rsync protocol and didn’t see one.

8

@garrett

I finally got around to test this - sorry for the wait.

It seems to be working as I would expect, but I did encounter some minor set up issues I wanted to share:

  • The user you login with using azcopy login needs to be assigned a role as Storage Blob Data Contributor or Storage Blob Data Owner for the container.
  • I initially didn’t specify root under the remote configuration, so the folder structure in the container were messed up, but I was still able to guild push. Trying to query the runs afterwards didn’t work though. If root is required, it would be nice to have a little check for that.

Other than that it is working flawlessly.

9

I’ll take a look at that root issue. In our tests root is specified. If that’s required, I’ll make sure it’s clear when it’s not provided.

10

@garrett

Ran into the root issue again :slight_smile: