> > Hi, > > We are replacing HDD with SSD, and we first (gradually) drain (reweight) the HDDs with 0.5 steps until 0 = empty. > > Works perfectly. > > Then (just for kicks) I tried reducing HDD weight from 3.6 to 0 in one large step. That seemed to have had more impact on the cluster, and we even noticed some OSD's temporarily go down after a few minutes. It all worked out, but the impact seemed much larger. Please clarify “impact”. Do you mean that client performance was decreased, or something else? > We never had OSDs go down when gradually reducing the weight step by step. This surprised us. Please also clarify what you mean by going down — do you mean being marked “down” by the mons, or the daemons actually crashing? I’m not being critical — I want to fully understand your situation. > Is it expected that the impact of a sudden reweight from 3.6 to 0 is bigger than a gradual step-by-step decrease? There are a lot of variables there, so It Depends. For sure going in one step means that more PGs will peer, which can be expensive. I’ll speculate, with incomplete information, that this is what most of what you’re seeing. > I would assume the impact to be similar, only the time it takes to reach HEALTH_OK to be longer. The end result, yes — the concern is how we get there. The strategy of incremental downweighting has some advantages: * If something goes wrong, you can stop without having a huge delta of data to move before health is restored * Peering is spread out * Impact on the network and drives *may* be less at a given time A disadvantage is that you end up moving some data more than once. This was worse with older releases and CRUSH details than with recent deployments. The impact due to data movement can be limited by lowering the usual recovery/backfill settings to 1 from their defaults, and depending on release by adjusting the osd_op_queue_cutoff. The impact due to peering can be limited by spreading out peering, either through an incremental process like yours, or by letting the balancer module do the work. There are other strategies as well, eg. disabling rebalancing, downweighting OSDs in sequence or a little at a time then enabling balancing when 0. > > Thanks, > MJ > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |