To take control of the influence of backfilling, I will set primary-affinity to 0 of those new OSDs and adjust backfilling configurations.
I change the crush weights. My 4 second sleep doesn't let peering finish for each one before continuing. I'd test with some small steps to get an idea of how much remaps when increasing the weight by $x. I've found my cluster is comfortable with +1 increases...also it take awhile to get to a weight of 11 if I did anything smaller._______________________________________________
for i in {264..311}; do ceph osd crush reweight osd.${i} 11.0;sleep 4;done
Kevin
On 7/24/19 12:33 PM, Xavier Trilla wrote:
Hi Kevin,
Yeah, that makes a lot of sense, and looks even safer than adding OSDs one by one. What do you change, the crush weight? Or the reweight? (I guess you change the crush weight, I am right?)
Thanks!
I often add 50+ OSDs at a time and my cluster is all NLSAS. Here is what I do, you can obviously change the weight increase steps to what you are comfortable with. This has worked well for me and my workloads. I've sometimes seen peering take longer if I do steps too quickly but I don't run any mission critical has to be up 100% stuff and I usually don't notice if a pg takes a while to peer.
Add all OSDs with an initial weight of 0. (nothing gets remapped)
Ensure cluster is healthy.
Use a for loop to increase weight on all news OSDs to 0.5 with a generous sleep between each for peering.
Let the cluster balance and get healthy or close to healthy.
Then repeat the previous 2 steps increasing weight by +0.5 or +1.0 until I am at the desired weight.
Kevin
On 7/24/19 11:44 AM, Xavier Trilla wrote:
Hi,
What would be the proper way to add 100 new OSDs to a cluster?
I have to add 100 new OSDs to our actual > 300 OSDs cluster, and I would like to know how you do it.
Usually, we add them quite slowly. Our cluster is a pure SSD/NVMe one, and it can handle plenty of load, but for the sake of safety -it hosts thousands of VMs via RBD- we usually add them one by one, waiting for a long time between adding each OSD.
Obviously this leads to PLENTY of data movement, as each time the cluster geometry changes, data is migrated among all the OSDs. But with the kind of load we have, if we add several OSDs at the same time, some PGs can get stuck for a while, while they peer to the new OSDs.
Now that I have to add > 100 new OSDs I was wondering if somebody has some suggestions.
Thanks!
Xavier.
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
