2 Things about this. First, Mons do not communicate over the cluster network. Only OSD daemons send any traffic over that network. Mons, MDS, RGW, MGR, etc all communicate over the public network. OSDs communicate with all clients, Mons, etc on the public network and with each other on the cluster network. That is for sharing extra replicas to other OSDs, backfilling, recovering, pinging each other to see if things are up and running. etc.
Second, What will happen if you change the cluster network on 1 node and not the others is that the other nodes will report to the Mons that every OSD on that node is down because they won't respond to pings and the 1 node will report to the Mons that every other OSD in the cluster is down. Making changes to the cluster network is easiest by stopping all OSDs and starting them back up with the new settings.
On Thu, Jun 7, 2018 at 3:46 AM Kevin Olbrich <ko@xxxxxxx> wrote:
Hi!_______________________________________________When we installed our new luminous cluster, we had issues with the cluster network (setup of mon's failed).We moved on with a single network setup.Now I would like to set the cluster network again but the cluster is in use (4 nodes, 2 pools, VMs).What happens if I set the cluster network on one of the nodes and reboot (maintenance, updates, etc.)?Will the node use both networks as the other three nodes are not reachable there?Both the MONs and OSDs have IPs in both networks, routing is not needed. This cluster is dualstack but we set ms_bind_ipv6 = true.Thank you.Kind regardsKevin
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
