Host key verification (known_hosts) with ProxyJump/ProxyCommand

[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]


Hi all,

I noticed a bit of an odd issue with maintaining `known_hosts` when the target machine is behind a bastion using `ProxyJump` or `ProxyCommand` with host key clashes.

Client for me right now is OpenSSH_9.3p1 on Gentoo Linux/AMD64. I'm a member of a team, and most of us use Ubuntu (yes, I'm a rebel). Another team who actually maintain this fleet often access the same machines via Windows 10/11 boxes (not sure if they use native OpenSSH or WSL). I rather suspect this issue actually is not platform-specific.

Target machines are using OpenSSH on Debian/ARMHF (the exact version varies with the exact OS version) -- hardware is essentially industrialised Raspberry Pis.

The bastions are typically OpenWRT-based (Teltonica) routers with Dropbear SSHd.

We share a configuration tree via a git repository which contains `Host` entries for each of the target machines and the intermediate bastion hosts.

The target machines are mostly using "private" address space in the subnet (although some are using addresses, because some goose thought all subnets were "private"). The bastion hosts run `dhcpd` and in many cases, the target acquires its address via DHCP (yes, super bad idea).

That means that in many cases, multiple _different_ OpenSSH servers, have the "same" IPv4 local address. Since when using `ProxyJump`, this address is recorded along side the host's public key in the user's `known_hosts` file, you can imagine when one logs into one server via one bastion, then tries to log in to a different server via a different bastion if that second server has the same local IPv4 address.

The crux of this is that we cannot assume the local IPv4 address is unique, since it's not (and in many cases, not even static).

In the case of `ProxyCommand`, the IPv4 address of the target may not even be obvious to the SSH client process.

-- Possible solutions / work-arounds using existing OpenSSH client --

I looked around for a solution, I ruled out turning off `StrictHostKeyChecking` (as seen on ServerFault) as a terrible idea asking for a Man-In-The-Middle attack.

DNS might "solve" the problem, but is likely to be messy to implement (bastion hosts are resource constrained, not sure if they do dynamic DNS for their LAN clients).

I know I'll get push-back from the other team if I try to mandate unique local IPv4 addresses. (This is the same team that unwittingly decided to rely on DHCP static assignment to "do the right thing".)

Link-local IPv6 is a tempting prospect: I have used this to get into a target node when its DHCP client has gone AWOL leaving IPv4 unconfigured, and being derived from the MAC address, *should* be globally unique, but this assumes a dual-stack LAN. (And the engineering team who look after these are likely to baulk at this. They barely understand IPv4!)

Port forwarding will require a lot of manual piss-farting around on the router's config webpage… and will likely break if the embedded DHCPd decides to not assign the static IP the target machine was supposed to get.

In the `ssh_config` man page, I see there is a `KnownHostsCommand` option, which could possibly be employed here, however since the files are "shared" by multiple users, there's the issue of paths, since I'll bet the `KnownHostsCommand` is relative to ${PWD} and not ~/.ssh/config or any config file imported by it.

User or Global `known_hosts` won't work due to the format of the file used (it assumes a unique endpoint IP address, which we know is not unique). (I have `HashKnownHosts` turned off on this Gentoo machine, my workplace laptop has it turned on due to Ubuntu's default. Not sure if this hash takes into account `ProxyJump` paths or `ProxyCommand` options.)

-- Possible solutions that require OpenSSH client changes --

One way that might work would be to embed the effective `ProxyJump`/`ProxyCommand` path in the "host name" stored in `known_hosts` -- will look ugly as sin, but at least the client can "uniquely" identify each server, and determine which key to use for validation.

e.g. you might have in known_hosts{ProxyJump user@,user2@} <algo> <key>{ProxyCommand user@ 22} <algo> <key>
the {} part encodes the path by which you reach the host.

Alternative might be an "ExpectHostKey" option that can be put in ~/.ssh/config or specified with "-o ExpectHostKey=…" that tells the SSH client "ignore your known_hosts file, the host *will* be using this key". So if you know the public key (e.g. you did a `ssh_keyscan`), you can either:

put in .ssh/config:

Host mytarget
	ProxyJump user2@bastion2
	ExpectHostKey ecdsa-sha2-nistp256 AAAA…=

Host bastion2
	ProxyJump user@bastion1
	ExpectHostKey ecdsa-sha2-nistp256 AAAA…=

Host bastion1
	ExpectHostKey ecdsa-sha2-nistp256 AAAA…=

OR, you might specify it on the command line (assuming the bastions are "known")

ssh -o ExpectHostKey="ecdsa-sha2-nistp256 AAAA…=" \
	-J user@bastion user@target

Bonus with this latter approach is that in a config sharing environment using a SCM (whether it be git, Subversion, CVS… whatever), assuming that repository was protected and "trusted", it would enable all members of a team to automatically "trust" the host key with minimal infrastructure set-up.

Are any of the above ideas feasible? Did I miss an obvious solution to this?
Stuart Longland (aka Redhatter, VK4MSL)

I haven't lost my mind...'s backed up on a tape somewhere.
openssh-unix-dev mailing list

[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [ECOS]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux