Hey Burkhard, thanks a lot for the insight. Especially knowing that mons/mgr don't use the cluster network is a good information. I discovered today as well that both public network and cluster network are actually lists of networks: https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/ even in Nautilus this seems already to be the case: https://docs.ceph.com/en/nautilus/rados/configuration/network-config-ref/ So what we will probably try the next days is to *add* the Kubernetes network ranges, which are already routed. Interestingly, checking on a rook/pacific installation, there is no public or cluster network configuration at all anymore, which makes me question, what these settings did actually do in the first place? Does setting public or cluster just limit the binds or the selection of target addresses and without the setting connections from anywhere are allowed? Sunny greetings from Switzerland, Nico p.s.: Sorry did not make it clear in the original post, cluster = public network in ur case. Burkhard Linke <Burkhard.Linke@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> writes: > Hi, > > > some years ago we changed our setup from a IPoIB cluster network to a > single network setup, which is a similar operation. > > > The OSD use the cluster network for heartbeats and backfilling > operation; both use standard tcp connection. There is no "global view" > on the networks involved; OSDs announce their public and private > network (if present) via an update to the OSD map on OSD boot. OSDs > expect to be able to create TCP connections to the announced IP > addresses and ports. Mon and mgr instances do not use the cluster > network at all. > > If you want to change the networks (either public or private), you > need to ensure that during the migration TCP connectivity between the > old networks and the new networks is possible, e.g. via a route on > some router. Since we had an isolated IPoIB networks without any > connections to some router, we used one of the ceph hosts as > router. Worked fine for a migration in live production ;-) > > Regarding the network size: I'm not sure whether the code requires an > exact CIDR match for the interface. If in doubt, have a look at the > source code > > As already mentioned in another answer, most setups do not require an > extra cluster network. It is extra effort both in setup, maintenance > and operating. Unless your network is the bottleneck you might want to > use this pending configuration change to switch to a single network > setup. > > > Regards, > > Burkhard Linke > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx -- Sustainable and modern Infrastructures by ungleich.ch _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |