Remotes Reference

Overview

This reference documents remote types, their supported attributes, and provides examples.

A remote type is defined using the type attribute. The types below are supported. Refer to the applicable section below for details.

type attribute
ssh
ec2
s3

SSH

SSH Attributes

description

Remote description (string)

host

Host name to use when connecting to the remote (required string)

port

Port for SSH connections (integer)

user

User for SSH connections (string)

private-key

Local path to the private key for SSH connections (string)

proxy

Proxy to use for the SSH connection (string)

connect-time

Number of seconds to wait for an SSH connection before quitting (integer)

venv-path

Path to a virtual environment on the remote host (string)

Guild activates the environment for each remote command.

Paths are relative to the remote user home directory.

Virtual environments are standard Python virtual environments that are created with virtualenv, guild init, or the Python venv module.

To strictly control the environment activation, use venv-activate instead.

guild-env

Alias for venv-path (see above)

conda-env

Conda environment name on the remote (string)

Guild activates the Conda environment for each remote command. If guild-env is also specified, the Guild environment is activated rather than the Conda environment.

venv-activate

Command used to activate an environment on the remote (string)

Use this to specify the command that Guild uses to activate an environment for remote commands. If this attribute is specified, Guild ignores guild-env and conda-env.

use-prerelease

Whether to use pre-release versions for required packages (boolean)

When Guild starts a remote run, it installs the run code along with required Python packages that are specified in the project package definition or in requirements.txt. Set this flag to yes to instruct Guild to install pre-release versions of required packages.

init

Shell command to run when the remote is reinitialized (string)

See SSH Start and Stop details information on initializing an SSH remote.

SSH Status

When you run guild remote status, Guild attempts to connect to the remote host over SSH using the specified connect information. If the connect succeeds, Guild considers the remote to be available.

SSH Start and Stop

SSH remotes cannot be started or stopped. Guild assumes that the specified host is available using the specified connect information.

SSH remotes can, however, be initialized using the --reinit option to guild remote start. When --reinit is specified, Guild runs the shell command defined in the remote init attribute.

Consider the following remote configuration:

remotes:
  my-remote:
    type: ssh
    host: remote-hostname
    init: |
      sudo apt update -y
      sudo apt upgrade -y

The command guild remote start my-remote --reinit runs the init command on the remote host. Use this to ensure that the remote is configured correctly before running Guild commands.

SSH Remote Commands

The commands below can be used with SSH remotes.

check Check Guild on the remote
run Run an operation on a remote
stop Stop runs in progress on a remote
watch Connect to a remote run in progress and watch its output
runs List runs on a remote
runs info Show information about a remote run
ls List remote run files
diff Diff remote runs
cat Show remote run file or output
label Apply a label to one or more remote runs
runs delete Delete remote runs
runs restore Restore deleted remote runs on a remote
runs purge Purge deleted remote runs on a remote
pull Copy remote runs to the local environment
push Copy local runs to the remote

SSH Security

Security is managed by the SSH protocol between the local system (client) and the remote (server). Connection settings are defined as attributes (see above) and by the SSH environment. Refer to the SSH documentation for your system for additional security details.

SSH Examples

Refer to Remotes Cheatsheet for SSH configuration examples.

EC2

EC2 provide the same features as SSH remotes. In addition, EC2 remotes can be started and stopped on EC2 by configuring EC2 instance attributes. Refer to Attributes below for details.

EC2 Attributes

Any SSH Attribute may be used to configure an EC2 remote. Refer to that section for details.

In addition to supporting SSH attributes, EC2 remotes support various EC2-specific settings. These are used when running guild remote start to create new EC2 resources to support remote operations.

ami

AMI used to create the EC2 instance (required string)

instance-type

Type of EC2 instance to create (required string)

region

AWS region to create the EC2 instance in (string)

If this value isn’t specified, Guild uses the value defined by the AWS_DEFAULT_REGION environment variable.

root-device-size

Size of the root volume created for the server (integer)

If this value is omitted, the default volume size for the AMI is used.

EC2 Status

On guild remote status, Guild attempts to connect to the remote host over SSH using the specified connect information. If the connect succeeds, Guild considers the remote to be available.

EC2 Start and Stop

On guild remote start, Guild uses Terraform to start an EC2 instance using the EC2 remote settings. On guild remote stop, Guild similarly terminates the EC2 instance using Terraform.

EC2 Remote Commands

The commands below can be used with EC2 remotes.

check Check Guild on the remote
run Run an operation on a remote
stop Stop runs in progress on a remote
watch Connect to a remote run in progress and watch its output
runs List runs on a remote
runs info Show information about a remote run
ls List remote run files
diff Diff remote runs
cat Show remote run file or output
label Apply a label to one or more remote runs
runs delete Delete remote runs
runs restore Restore deleted remote runs on a remote
runs purge Purge deleted remote runs on a remote
pull Copy remote runs to the local environment
push Copy local runs to the remote

EC2 Security

The following environment variables must be defined when starting or stopping an EC2 remote.

AWS_ACCESS_KEY_ID The AWS access key ID that has the requisite permissions for starting or terminating the applicable EC2 instance.
AWS_SECRET_ACCESS_KEY The AWS secret access key associated with the access key ID.

The following environment variables may be optionally defined:

AWS_DEFAULT_REGION The AWS region the remote instance is created in. This value may alternatively be defined in the EC2 remote configuration.

Once an EC2 remote is created, security is managed by the SSH protocol between the local system (client) and the remote (server).

EC2 Examples

Refer to Remotes Cheatsheet for EC2 configuration examples.

S3

Use S3 remotes to store runs remotely. S3 buckets support the full set of remote management commands. See Remote Commands below for details.

S3 Attributes

description

Remote description (string)

bucket

S3 bucket to store runs in (required string)

root

Path in the S3 bucket to store runs in (string)

region

AWS region where the bucket was created (string)

The region may alternatively be specified using the AWS_DEFAULT_REGION environment variable when running a remote command for the S3 remote.

S3 Status

When you run guild remote status, Guild checks that the bucket exists and that Guild can read from it. This check does not perform a write test.

S3 Start and Stop

S3 remotes can be started and stopped. When started, Guild creates the remote bucket as needed. When stopped, Guild deletes the remote bucket if it exists.

Important Stopping an S3 remote will delete all contents of the remote bucket, including Guild runs any objects that were copied to the bucket independently of Guild. This operation cannot be undone.

S3 Remote Commands

The following commands can be used with S3 remotes.

runs List runs on a remote
runs info Show information about a remote run
runs delete Delete remote runs
runs restore Restore deleted remote runs on a remote
runs purge Purge deleted remote runs on a remote
pull Copy remote runs to the local environment
push Copy local runs to the remote

S3 Security

The following environment variables must be defined when running any command on an S3 remote.

AWS_ACCESS_KEY_ID The AWS access key ID that has the requisite permissions on the remote S3 bucket.
AWS_SECRET_ACCESS_KEY The AWS secret access key associated with the access key ID.

The following environment variables may be optionally defined:

AWS_DEFAULT_REGION The AWS region associated with the bucket. This value may alternatively be defined in the S3 remote configuration.

S3 Examples

Refer to Remotes Cheatsheet for S3 configuration examples.