> Op 31 juli 2017 om 11:40 schreef Stuart Longland <stuartl@xxxxxxxxxxxxxxxxxx>: > > > On 31/07/17 19:10, Wido den Hollander wrote: > > > >> Op 30 juli 2017 om 2:42 schreef Stuart Longland <stuartl@xxxxxxxxxxxxxxxxxx>: > >> As a result, I see messages like this from clients: > >>> oneadmin@opennebula:~$ rados df --id libvirt > >>> 2017-07-30 09:58:32.389376 7f4f611b4700 0 -- :/3981532287 >> [2001:44b8:21ac:70fc::2]:6789/0 pipe(0x560bbfd3eb80 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x560bbfd3fe60).fault > >> > >> It may bounce around a few times before it gives up on IPv6 and reverts > >> back to IPv4. I tried setting up xinetd to proxy the V6 connection to > >> V4, but then ceph-mon refuses the connection from xinetd. > >> > >> My only work-around has been to change the clients to use IPv4 addresses > >> instead of DNS hostnames, which is distasteful at best. > > > > See my information below, but you should create a DNS record with only A or AAAA-records, not mixed. > > Creating DNS records with both has been standard practice for some time > now and I do it just fine with great success with HTTP and SSH servers. > Yes, but that's SSH and HTTP, not Ceph. For Ceph it just works differently. I recommend that you create ONE DNS record, like: ceph-mon.local. AAAA 2001:db8::100 ceph-mon.local. AAAA 2001:db8::101 ceph-mon.local. AAAA 2001:db8::102 Give 'ceph-mon.local' to your clients and they fill figure it out. > >> I've read the documentation, and while there appears to be a flag to > >> make OSDs bind to IPv6, there doesn't seem to be an option for monitors. > >> > > > > ms_bind_ipv6 = true > > > > But keep in mind, that MONs will ONLY bind the the address specified on the MONMap. > > > > A Ceph cluster can't run Dual-Stack, so it has to run either IPv4 OR IPv6. > > So what do I do if some clients are IPv4 only, and some IPv6 only? > The clients need to have the same address family available as the cluster runs. Internally it can talk either v4 OR v6, not both. Not to clients either. > >> Given that it all seems to be TCP communications, I'm surprised we can't > >> just bind a socket using AF_INET6 and support both simultaneously. Done > >> that in the past, and it has worked well. > >> > >> Anyway, is there something I missed or has this been overlooked? > > > > No, this is on purpose. Ceph relies heavily on OSDMaps and MONMaps which contain IPs internally. > > > > You'd have to choose if you want to run Ceph over v4 or v6, but it's not possible to mix this. > > IP addresses to hosts is not a 1:1 mapping. A host may have more than > one IP address, even in the same address family and on the one interface. > > Knowing that fact, shouldn't Ceph just record *all* the addresses seen > for a hostname as aliases of that one host? > No, each OSD only has a single field in the OSDMap where it's address is stored. Same goes for the MONs. There have been talks about making this dual-stack, but there are some issues around this. For now it's not possible to do that. > i.e. > > mon.a: 10.128.45.1:6789, [2001:db8::1]:6789 > mon.b: 10.128.45.2:6789, [2001:db8::2]:6789 > mon.c: 10.128.45.3:6789, [2001:db8::3]:6789 > > Then, it wouldn't matter, and moreover, it'd be possible to change a > monitor node's IP address quite easily that way: you'd just add the new > IP as another IP alias, then remove the old one. (I do accept the > argument that monitor nodes should not change IP address… but sometimes > circumstance makes this necessary.) > OSD IPs are dynamic anyway. When they boot their IP is dynamically recorded to the OSDMap and distributed inside the cluster. So you can swap IPs as fast as you want. Wido > Currently, I think the docs mention doing similar, but effectively > decommissioning then re-commissioning the monitor. > -- > Stuart Longland (aka Redhatter, VK4MSL) > > I haven't lost my mind... > ...it's backed up on a tape somewhere. > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |