On 31/07/17 19:10, Wido den Hollander wrote: > >> Op 30 juli 2017 om 2:42 schreef Stuart Longland <stuartl@xxxxxxxxxxxxxxxxxx>: >> As a result, I see messages like this from clients: >>> oneadmin@opennebula:~$ rados df --id libvirt >>> 2017-07-30 09:58:32.389376 7f4f611b4700 0 -- :/3981532287 >> [2001:44b8:21ac:70fc::2]:6789/0 pipe(0x560bbfd3eb80 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x560bbfd3fe60).fault >> >> It may bounce around a few times before it gives up on IPv6 and reverts >> back to IPv4. I tried setting up xinetd to proxy the V6 connection to >> V4, but then ceph-mon refuses the connection from xinetd. >> >> My only work-around has been to change the clients to use IPv4 addresses >> instead of DNS hostnames, which is distasteful at best. > > See my information below, but you should create a DNS record with only A or AAAA-records, not mixed. Creating DNS records with both has been standard practice for some time now and I do it just fine with great success with HTTP and SSH servers. >> I've read the documentation, and while there appears to be a flag to >> make OSDs bind to IPv6, there doesn't seem to be an option for monitors. >> > > ms_bind_ipv6 = true > > But keep in mind, that MONs will ONLY bind the the address specified on the MONMap. > > A Ceph cluster can't run Dual-Stack, so it has to run either IPv4 OR IPv6. So what do I do if some clients are IPv4 only, and some IPv6 only? >> Given that it all seems to be TCP communications, I'm surprised we can't >> just bind a socket using AF_INET6 and support both simultaneously. Done >> that in the past, and it has worked well. >> >> Anyway, is there something I missed or has this been overlooked? > > No, this is on purpose. Ceph relies heavily on OSDMaps and MONMaps which contain IPs internally. > > You'd have to choose if you want to run Ceph over v4 or v6, but it's not possible to mix this. IP addresses to hosts is not a 1:1 mapping. A host may have more than one IP address, even in the same address family and on the one interface. Knowing that fact, shouldn't Ceph just record *all* the addresses seen for a hostname as aliases of that one host? i.e. mon.a: 10.128.45.1:6789, [2001:db8::1]:6789 mon.b: 10.128.45.2:6789, [2001:db8::2]:6789 mon.c: 10.128.45.3:6789, [2001:db8::3]:6789 Then, it wouldn't matter, and moreover, it'd be possible to change a monitor node's IP address quite easily that way: you'd just add the new IP as another IP alias, then remove the old one. (I do accept the argument that monitor nodes should not change IP address… but sometimes circumstance makes this necessary.) Currently, I think the docs mention doing similar, but effectively decommissioning then re-commissioning the monitor. -- Stuart Longland (aka Redhatter, VK4MSL) I haven't lost my mind... ...it's backed up on a tape somewhere.
Attachment:
signature.asc
Description: OpenPGP digital signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
