Exporting guild runs from remote sever

Troubleshooting
1

Hi Garret and co,
I have used guild on my last project and liked it! This time, I am running my scripts on different remote servers. I would like to import the runs to my desktop to view them all together. I am not sure how to do this. I would prefer a solution that doesnt involve the remote configuration, because I dont know much about it and my system requires often to enter passwords.
Thanks for the help!

2

Hi Shannon! I’m glad you’re enjoying Guild.

To view runs that are on one machine from another machine, you have a few options:

  • Connect to the remote server over a terminal (generally ssh) and use Guild’s terminal based commands to view runs. This can get you surprisingly far as you can get to all run info including generated files over a terminal. The notable exception is that the open command is limited to terminal based commands. E.g. you can’t run open on a remote server over ssh to view a using a windowing app running on your desktop.

  • Run guild view on the remote server on a port that is available to you (ports below 1024 will require root privilege) and that you can access from your desktop. This often requires opening ports in firewalls. This is the sort of problem you have running any web based app on a remote server — e.g. TensorBoard, etc.

  • Use an HTTP server running on your remote server to make a directory available for browsing. A simple way to do this is to run python -m http.server <port> from a directory on the remote server. Guild runs will be under $GUILD_HOME/runs. Each run is saved in a unique directory that corresponds to the run ID. Then, from your local system, navigate to the remote host + port and you should see the directory listing, which you can browse to view/download files, etc. Note that this approach is all low level file access. In a pinch this is a good way to look at files remotely.

  • Use a shared network file system like NFS or Samba.

  • Copy the runs from the remote server to your local system.

In the case of copy, you can use Guild’s pull command, but this requires a remote config, which you’ve said you’d prefer to avoid. I think this is the most robust approach of the alternatives.

  • You get a local copy of the runs, so you have full access to the programs installed on your system.

  • You’re not relying on network based file systems, which are notoriously fickle — IO errors and poor performance.

  • Other than user auth (ssh) there’s nothing to configure — e.g. no servers to run, etc.

If you’re concerned about the need to enter a password all the time, you can setup passwordless auth for ssh. This can be a bit technical, but once you have it configured, it’s quite streamlined. The only maintenance overhead you have when using new remote servers is to include your public ssh key in your remote user’s ~/.ssh/authorized_keys file.

If you prefer to copy runs yourself, you can use any standard file copy protocol such as scp, rsync, or ftp. FTP can be configured to avoid password auth but honestly there’s more work there than setting up ssh private key access (see link above a list of resources).

To manually copy runs, copy $GUILD_HOME/runs from the remote to local. Depending on the configuration, you can initiate the copy from your local system or from the remote (pushing runs from remote to your local system).

1 Like
3

I should note — as you asked specifically about exporting — that the export command applies to local runs, not remote. Use export to select specific runs from your environment and copy or move them to a directory. This is useful for archiving or backup.

To get runs from a different system, the pull command is what you’d use, assuming you go through Guild rather than copy runs yourself.

You could use export on the remote server to copy/move runs to a directory, which you can then copy yourself to some location. E.g. you could then use s3cmd to copy the exported runs to S3, where you could access them from your local machine. That said, I’d recommend using Guild push with an S3 remote for this.

Using S3 this way you can implement a “runs sink” where you push all runs from any server. With runs consolidated in a single location, you can pull whatever you need from any other server. This is a nice pattern as you get both a backup of all runs as well as a scheme for sharing across any number of users and systems.

1 Like
4

Great I think the manual option is fine for me !