On Mon, Jan 28, 2019 at 4:00 AM Sage Weil <sweil@xxxxxxxxxx> wrote: > > msgr1 has some super cludgey behavior that's used to detect what IP > address the client or daemon should identify as. If the daemon hasn't > explicitly binded to a specific IP (i.e., it bound to 0.0.0.0, [::], or > didn't bind at all) then the first time it connects to another peer the > peer will send the IP we appear to be connecting from in the initial > banner. > > It seems to have worked out mostly okay, but it's definitely a bit weird. > The first connection is always to the monitor, so this means that the IP > that an OSD or MDS daemon uses is always the one that on the same network > as the monitor (or whichever IP the kernel decides to use to route to it). > > A side-effect of this is that, in theory, a client that is behind NAT > could connect to a ceph cluster. It will end up being identified by the > NATed IP that the cluster sees and the random 64-bit nonce. > > Note that (to my knowledge) this has never been tested, so it only > theoretically works.. > > The initial msgr2 implementation simplifies this by instead calling > getsockname(2) on the first outgoing connection to see what IP we're > connecting from. That removes the weird dependency on the other end tell > us who we are, but it means that NAT won't work. > > So... should we try to make the NAT scenario work in msgr2? > > We can do it with a minor-ish change to have the accepting end share our > apparent IP sooner in teh exchange (probably after the initial banner). > (The current code shares it as part of the server_ident, but that's too > late in the exchange to serve the same role it did in msgr1.) > > sage I know RBD has some test cases where is runs librbd clients *within* a QEMU VM via the built-in host NATing (e.g. OpenStack devstack test cases). Is there some reason why the OSDs care about the real IP address of the client? -- Jason
