On Mon, Aug 29, 2022 at 12:49 AM Burkhard Linke <Burkhard.Linke@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> wrote: > > Hi, > > > some years ago we changed our setup from a IPoIB cluster network to a > single network setup, which is a similar operation. > > > The OSD use the cluster network for heartbeats and backfilling > operation; both use standard tcp connection. There is no "global view" > on the networks involved; OSDs announce their public and private network > (if present) via an update to the OSD map on OSD boot. OSDs expect to be > able to create TCP connections to the announced IP addresses and ports. > Mon and mgr instances do not use the cluster network at all. > > If you want to change the networks (either public or private), you need > to ensure that during the migration TCP connectivity between the old > networks and the new networks is possible, e.g. via a route on some > router. Since we had an isolated IPoIB networks without any connections > to some router, we used one of the ceph hosts as router. Worked fine for > a migration in live production ;-) To be a little more explicit about this: Ceph stores the IP addresses of live OSDs and MDSes in their respective cluster maps, but otherwise does not care about them at all — they are updated to the daemon's current IP on every boot. The monitor IP addresses are fixed identities, so moving them requires either adding new monitors and removing old ones, or else doing surgery on their databases to change the IPs by editing the monmap they store (and then updating the local config for clients and OSDs so they point to the new locations and can find them on bootup.) But for the OSDs etc, you're really just worried about the local configs or your deployment tool. (And, depending on how you arrange things, you may need to take care to avoid the OSDs moving into new CRUSH buckets and migrating all their data.) -Greg > > Regarding the network size: I'm not sure whether the code requires an > exact CIDR match for the interface. If in doubt, have a look at the > source code > > As already mentioned in another answer, most setups do not require an > extra cluster network. It is extra effort both in setup, maintenance and > operating. Unless your network is the bottleneck you might want to use > this pending configuration change to switch to a single network setup. > > > Regards, > > Burkhard Linke > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |