key file is not used if other keys exist in ssh agent

[xinfengliu]

While debugging a k0sctl issue, I found my specified keyPath for ssh is not used and ssh connection failed.

spec:
  hosts:
  - ssh:
      address: rocky9-0
      keyPath: /home/docker/.ssh/id_ed25519
      port: 22
      user: root
...

To work around the issue, I had to manually add the key to ssh agent or unset SSH_AUTH_SOCK or clearing keys in ssh agent.

I reviewed k0sproject/rig codes and found the reason is that I have already had other ssh keys in SSH agent, and the specified ssh key file is configured as another ssh.AuthMethod in https://github.com/k0sproject/rig/blob/v0.19.0/ssh.go#L434 , so ssh.ClientConfig.Auth has two AuthMethod, both method() is publickey , golang.org/x/crypto/ssh client_auth.go only tries the first one for the same auth method (https://cs.opensource.google/go/x/crypto/+/refs/tags/v0.29.0:ssh/client_auth.go;l=101), so the specified ssh key file is never used.

[kke]

I think one problem may be if /home/docker/.ssh/id_ed25519 is the default key path, in which case rig could assume that an explicit keypath wasn't given and tries other methods 🤔 If a keypath is given, it should only try that key and nothing else.

[xinfengliu]

I tried non-default key path, but the problem remains same. I added a few debug logs, here's ~/.cache/k0sctl/k0sctl.log

time="24 Dec 24 12:05 CST" level=debug msg="retrying, attempt 8 - last error: not connected: client connect: ssh dial: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain"
time="24 Dec 24 12:05 CST" level=info msg="### SSH: [ssh] rocky9-0:22"
time="24 Dec 24 12:05 CST" level=debug msg="using SSH_AUTH_SOCK=/tmp/ssh-XXXXcro9PG/agent.1283978"
time="24 Dec 24 12:05 CST" level=debug msg="[ssh] rocky9-0:22: using all keys (2) from ssh agent because a keypath was not explicitly given"
time="24 Dec 24 12:05 CST" level=info msg="## config.Auth: [0x8f7c60]"
time="24 Dec 24 12:05 CST" level=info msg="## keyPath: /home/docker/test/mke4/id_ed25519"
time="24 Dec 24 12:05 CST" level=info msg="### keyPaths: [/home/docker/test/mke4/id_ed25519]"
time="24 Dec 24 12:05 CST" level=info msg="### config.Auth: [0x8f7c60 0x8f9120]"
time="24 Dec 24 12:05 CST" level=debug msg="[SSH] rocky9-0: failed to connect: ssh dial: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain"

[cgroschupp]

From https://cs.opensource.google/go/x/crypto/+/refs/tags/v0.38.0:ssh/client.go;l=210:

// Auth contains possible authentication methods to use with the
// server. Only the first instance of a particular RFC 4252 method will
// be used during authentication.
Auth []AuthMethod

corresponds to the documentation.

As a workaround, i use this: env -u SSH_AUTH_SOCK k0sctl

[cgroschupp]

@xinfengliu can you please try it with my PR.

[xinfengliu]

@cgroschupp
Sorry I don't work anymore and don't have computing resources to verify your PR. Please ask the project owner to review. Thanks.

[cgroschupp]

@kke I found the problem. Could you please review my PR.

[apreiml]

@cgroschupp fixes part of the problem. But another issue is that the keys provided by the key path are tried after all the agent keys. I have too many ssh keys in my agent and this triggers a too many authentication tries error on some of my machines, which prevents me to use k0sctl

I'd like to propose to prefer the ones provided explicitly. I would go so far as to not include any other agent keys, if a key is provided.

I could prepare a pull request once the fix of @cgroschupp is merged.