Guild pull no matching run found

Hi,

I am having problems with pulling runs from a remote server.

guild runs --remote remote

[1:5c07e941]  srl/state:train-rl  2020-06-22 14:45:06  completed  env=SingleGoal-v01

When I try to pull the run it tells me there is no matching run was found:

guild pull remote 5c07e941 

Getting remote run info 
guild: could not find a run matching '5c07e941'

My config.yml looks as follows:

remotes:
  remote:
    type: ssh
    host: remote
    user: username
    guild-home: /home/username/guild
    guild-env: /home/username/guild/venv

The guild version I use is 0.7.0.rc11.
Pushing a run to the remote works fin.

Have I done something wrong or is it a bug?

Kind regards,
Bart

Hello and welcome! I’m sorry you’re facing this issue. This certainly appears buggy. I’m not able to easily replicate this on rc11 so there may be something specific to your case.

A few questions:

  • What happens when you use the full run ID (you can get that by including the --verbose flag to guild runs)
  • What happens when you use the run index (i.e. 1 in this the example above, but it’ll be different if there are new runs)
  • Can you pull other runs using the short run ID? If you only have the one run, generate one more — something simple is fine.
  • If you run guild pull remote (no run args) does Guild pull the runs as expected?

Eh, sorry - hold of on that list of TODOs for a moment.

Where are your runs located on this remote? Based on the config, they should be in /home/username/guild/runs. But Guild will setup your environment so they’re under /home/username/guild/venv/.guild/runs.

Try deleting guild-home attr (or comment it out) from your user config. In the meantime I’ll setup something with that particular config and see if I can replicate this.

Your list from guild runs should match what pull sees — that it doesn’t here I think indicates a bug somewhere.

1 Like

Hey Garrett,

Thanks for you quick response.

Removing the guild-home attribute solved the problem.

Cheers,
Bart

Great!

I tried to recreate the issue where guild runs -r <remote> sees something different from guild pull <remote> that might explain what you saw above — I can’t recreate it. Is it possible that you modified your config in between running the runs list and pull commands?

For posterity, to fill in some information on the difference between guild-env and guild-home:

  • guild-env is used to activate a virtual environment whenever running on the remote host. You can see what’s going on under the covers by including the --debug option to guild (e.g. guild --debug run -r remote)
  • Unless guild-home is specified, Guild assumes <guild-env>/.guild on the remote host.
  • You can set guild-home independently of guild-env — this just means that runs live in a different location. guild-env is still used to activate the virtual environment.

No I did not modify the config in between running the the runs list and pull commands.

I added the guild-home = /home/username/guild attribute and re-ran the commands using the --debug flag. The guild pull ssh command does not have the export GUILD_HOME=/home/username/guild; attribute whereas the guild runs -r remote does:

guild --debug pull remote

Getting remote run info
DEBUG: [guild.remotes.ssh_util] ssh cmd: ['ssh', '-oStrictHostKeyChecking=no', 'username@remote', 'set -e; source /home/username/guild/venv/bin/activate; guild runs list --json'] 
guild --debug runs -r remote

DEBUG: [guild.remotes.ssh_util] ssh cmd: ['ssh', '-oStrictHostKeyChecking=no', 'username@remote', 'set -e; source /home/username/guild/venv/bin/activate; export COLUMNS=80; export GUILD_HOME=/home/username/guild; guild runs list '] 

What you’re seeing is behavior from an older version of Guild. Could you double check your local version of Guild?

guild --version

The remote version is not in play for this behavior.

That was it.

I installed the latest pre-release but had not deactivated and sourced the environment afterwards. Now everything is working correctly (also with the guild-home attribute).

Sorry for the trouble :sweat_smile:.

Bart

Haha no worries. Glad everything’s working!