Dear Maarten, For a cluster that size, I would not immediately enable the autoscaler but first enabled it in "warn" mode to sanity check what it would plan to do: # ceph osd pool set <pool> pg_autoscale_mode warn Please share the output of "ceph osd pool autoscale-status" so we can help guide what you do next. Also, you should be aware that there are some rare but unpleasant bugs that may be related to PG splitting (autoscaling). See https://tracker.ceph.com/issues/53729 You may want to wait until that issue is resolved before permanently enabling the autoscaler. Best Regards, Dan > On 02/07/2022 12:31 PM Maarten van Ingen <maarten.vaningen@xxxxxxx> wrote: > > > Hi, > > We are about to enable the PG autoscaler on CEPH. Currently we are running the latest subrelease of Nautilus with Bluestore and LVM. The current status of the autoscaler is that it’s turned off on all pools and the module is enabled. > > To make sure we do not kill anything, performance and/or data. I’d like some advice on how to proceed. > > We have about 11PiB of raw HDD storage and 40ish% is in use and about 550TiB of NVMe storage. In total we have about 1250 OSD’s of which about 300 are NVMe only OSD’s. We have crush rules to allow for NVMe only storage pools and HDD only storage pools > > For every pool we have we have set a target-size to guide the autoscaler a bit and also we have set a minimum of 256 PG’s per pool. > What now happens is that a few pools will have their amount of PG’s changed ranging from 4x to 16x. We have never changed the amount of PG’s in a pool with these factors (no more than 2x in a single go) and also with a lot less data. So we have no clear idea of what will happen when we enable the autoscaler. > > For example, one pool which has about 1PiB of user data will grow from 4k PG’s to 16k PG’s. This of course will involve a lot of data movement. Another pool with 100TiB of data will grow from 512 to 8k PG’s > > All pools are set with a size of 3 and thus the abovementioned 1PiB is 3PiB of raw data, we currently have no erasure coding pools. > > Can somebody help me out on how to proceed on a safe way to enable the autoscaler or tell me of it’s OK just to enable the autoscaler. We will enable it per pool to limit affected pools. > > Met vriendelijke groet, > > Kind Regards, > Maarten van Ingen > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |