Thanks for the reply Wido! How do you handle IPv6 routes and routing with IPv6 on public and cluster networks? You mentioned that your cluster network is routed, so they will need routes to reach the other racks. But you can't have more than 1 default gateway. Are you running a routing protocol to handle that?
We're using classless static routes via DHCP on v4 to solve this problem, and I'm curious what the v6 SLAAC equivalent was.
Thanks,
-richard
On Tue, Mar 28, 2017 at 8:30 AM, Wido den Hollander <wido@xxxxxxxx> wrote:
> Op 27 maart 2017 om 21:49 schreef Richard Hesse <richard.hesse@xxxxxxxxxx>:
>
>
> Has anyone run their Ceph OSD cluster network on IPv6 using SLAAC? I know
> that ceph supports IPv6, but I'm not sure how it would deal with the
> address rotation in SLAAC, permanent vs outgoing address, etc. It would be
> very nice for me, as I wouldn't have to run any kind of DHCP server or use
> static addressing -- just configure RA's and go.
>
Yes, I do in many clusters. Works fine! SLAAC doesn't generate random addresses which change over time. That's a feature called 'Privacy Extensions' and is controlled on Linux by:
- net.ipv6.conf.all.use_tempaddr
- net.ipv6.conf.default.use_tempaddr
- net.ipv6.conf.X.use_tempaddr
Set this to 0 and the kernel will generate one address based on the MAC-Address (EUI64) of the interface. This address is stable and will not change.
I like this very much as I don't have any static or complex network configurations on the hosts. It moves the whole responsibility of networking and addresses to the network. A host just boots and obtains a IP.
The OSDs contact the MONs on boot and they will tell them their address. OSDs do not need a fixed address for Ceph.
However, using SLAAC without Privacy Extensions means that in practice the address will not change of a machine, so you don't need to worry about it that much.
The biggest system I have running this way is 400 nodes running IPv6-only. 10 racks, 40 nodes per rack. Each rack has a Top-of-Rack switch running in Layer 3 and a /64 is assigned per rack.
Layer 3 routing is used between the racks that based on the IPv6 address we can even determine in which rack the host/OSD is.
Layer 2 domains don't expand over racks which makes a rack a true failure domain in our case.
Wido
> On that note, does anyone have any experience with running ceph in a mixed
> v4 and v6 environment?
>
> Thanks,
> -richard
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
