No repair IO and misplaced objects increasing with norebalance and nobackfill set. Thank you, Ray ________________________________ From: Ray Cunningham <ray.cunningham@xxxxxxxxxxxxxx> Sent: Wednesday, April 13, 2022 10:38:29 AM To: Dan van der Ster <dvanders@xxxxxxxxx> Cc: ceph-users@xxxxxxx <ceph-users@xxxxxxx> Subject: Re: Stop Rebalancing All pools have gone backfillfull. Thank you, Ray Cunningham Systems Engineering and Services Manager keepertechnology<http://www.keepertech.com/> (571) 223-7242 ________________________________ From: Ray Cunningham Sent: Wednesday, April 13, 2022 10:15:56 AM To: Dan van der Ster <dvanders@xxxxxxxxx> Cc: ceph-users@xxxxxxx <ceph-users@xxxxxxx> Subject: RE: Stop Rebalancing Perfect timing, I was just about to reply. We have disabled autoscaler on all pools now. Unfortunately, I can't just copy and paste from this system... `ceph osd pool ls detail` only 2 pools have any difference. pool1: pgnum 940, pgnum target 256, pgpnum 926 pgpnum target 256 pool7: pgnum 2048, pgnum target 2048, pgpnum883, pgpnum target 2048 ` ceph osd pool autoscale-status` Size is defined target size is empty Rate is 7 for all pools except pool7, which is 1.3333333730697632 Raw capacity is defined Ratio for pool1 is .0177, pool7 is .4200 and all others is 0 Target and Effective Ratio is empty Bias is 1.0 for all PG_NUM: pool1 is 256, pool7 is 2048 and all others are 32. New PG_NUM is empty Autoscale is now off for all Profile is scale-up We have set norebalance and nobackfill and are watching to see what happens. Thank you, Ray -----Original Message----- From: Dan van der Ster <dvanders@xxxxxxxxx> Sent: Wednesday, April 13, 2022 10:00 AM To: Ray Cunningham <ray.cunningham@xxxxxxxxxxxxxx> Cc: ceph-users@xxxxxxx Subject: Re: Stop Rebalancing One more thing, could you please also share the `ceph osd pool autoscale-status` ? On Tue, Apr 12, 2022 at 9:50 PM Ray Cunningham <ray.cunningham@xxxxxxxxxxxxxx> wrote: > > Thank you Dan! I will definitely disable autoscaler on the rest of our pools. I can't get the PG numbers today, but I will try to get them tomorrow. We definitely want to get this under control. > > Thank you, > Ray > > > -----Original Message----- > From: Dan van der Ster <dvanders@xxxxxxxxx> > Sent: Tuesday, April 12, 2022 2:46 PM > To: Ray Cunningham <ray.cunningham@xxxxxxxxxxxxxx> > Cc: ceph-users@xxxxxxx > Subject: Re: Stop Rebalancing > > Hi Ray, > > Disabling the autoscaler on all pools is probably a good idea. At least until https://tracker.ceph.com/issues/53729 is fixed. (You are likely not susceptible to that -- but better safe than sorry). > > To pause the ongoing PG merges, you can indeed set the pg_num to the current value. This will allow the ongoing merge complete and prevent further merges from starting. > From `ceph osd pool ls detail` you'll see pg_num, pgp_num, pg_num_target, pgp_num_target... If you share the current values of those we can help advise what you need to set the pg_num to to effectively pause things where they are. > > BTW -- I'm going to create a request in the tracker that we improve the pg autoscaler heuristic. IMHO the autoscaler should estimate the time to carry out a split/merge operation and avoid taking one-way decisions without permission from the administrator. The autoscaler is meant to be helpful, not degrade a cluster for 100 days! > > Cheers, Dan > > > > On Tue, Apr 12, 2022 at 9:04 PM Ray Cunningham <ray.cunningham@xxxxxxxxxxxxxx> wrote: > > > > Hi Everyone, > > > > We just upgraded our 640 OSD cluster to Ceph 16.2.7 and the resulting rebalancing of misplaced objects is overwhelming the cluster and impacting MON DB compaction, deep scrub repairs and us upgrading legacy bluestore OSDs. We have to pause the rebalancing if misplaced objects or we're going to fall over. > > > > Autoscaler-status tells us that we are reducing our PGs by 700'ish which will take us over 100 days to complete at our current recovery speed. We disabled autoscaler on our biggest pool, but I'm concerned that it's already on the path to the lower PG count and won't stop adding to our misplaced count after drop below 5%. What can we do to stop the cluster from finding more misplaced objects to rebalance? Should we set the PG num manually to what our current count is? Or will that cause even more havoc? > > > > Any other thoughts or ideas? My goals are to stop the rebalancing temporarily so we can deep scrub and repair inconsistencies, upgrade legacy bluestore OSDs and compact our MON DBs (supposedly MON DBs don't compact when you aren't 100% active+clean). > > > > Thank you, > > Ray > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an > > email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |