On 7/24/19 7:15 PM, Kevin Hrpcek wrote: > I often add 50+ OSDs at a time and my cluster is all NLSAS. Here is what > I do, you can obviously change the weight increase steps to what you are > comfortable with. This has worked well for me and my workloads. I've > sometimes seen peering take longer if I do steps too quickly but I don't > run any mission critical has to be up 100% stuff and I usually don't > notice if a pg takes a while to peer. > > Add all OSDs with an initial weight of 0. (nothing gets remapped) > Ensure cluster is healthy. > Use a for loop to increase weight on all news OSDs to 0.5 with a > generous sleep between each for peering. > Let the cluster balance and get healthy or close to healthy. > Then repeat the previous 2 steps increasing weight by +0.5 or +1.0 until > I am at the desired weight. That works, but with newer releases I've been doing this: - Make sure cluster is HEALTH_OK - Set the 'norebalance' flag (and usually nobackfill) - Add all the OSDs - Wait for the PGs to peer. I usually wait a few minutes - Remove the norebalance and nobackfill flag - Wait for HEALTH_OK Wido > > Kevin > > On 7/24/19 11:44 AM, Xavier Trilla wrote: >> >> Hi, >> >> >> >> What would be the proper way to add 100 new OSDs to a cluster? >> >> >> >> I have to add 100 new OSDs to our actual > 300 OSDs cluster, and I >> would like to know how you do it. >> >> >> >> Usually, we add them quite slowly. Our cluster is a pure SSD/NVMe one, >> and it can handle plenty of load, but for the sake of safety -it hosts >> thousands of VMs via RBD- we usually add them one by one, waiting for >> a long time between adding each OSD. >> >> >> >> Obviously this leads to PLENTY of data movement, as each time the >> cluster geometry changes, data is migrated among all the OSDs. But >> with the kind of load we have, if we add several OSDs at the same >> time, some PGs can get stuck for a while, while they peer to the new OSDs. >> >> >> >> Now that I have to add > 100 new OSDs I was wondering if somebody has >> some suggestions. >> >> >> >> Thanks! >> >> Xavier. >> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |