Hi, I see that you are using rgw RGW comes with many pools, yet most of them are used for metadata and configuration, those do not store many data Such pools do not need more than a couple PG, each (I use pg_num = 8) You need to allocate your pg on pool that actually stores the data Please do the following, to let us know more: Print the pg_num per pool: for i in $(rados lspools); do echo -n "$i: "; ceph osd pool get $i pg_num; done Print the usage per pool: ceph df Also, instead of doing a "ceph osd reweight-by-utilization", check out the balancer plugin : http://docs.ceph.com/docs/mimic/mgr/balancer/ Finally, in nautilus, the pg can now upscale and downscale automaticaly See https://ceph.com/rados/new-in-nautilus-pg-merging-and-autotuning/ On 04/30/2019 06:34 PM, Shain Miley wrote: > Hi, > > We have a cluster with 235 osd's running version 12.2.11 with a > combination of 4 and 6 TB drives. The data distribution across osd's > varies from 52% to 94%. > > I have been trying to figure out how to get this a bit more balanced as > we are running into 'backfillfull' issues on a regular basis. > > I've tried adding more pgs...but this did not seem to do much in terms > of the imbalance. > > Here is the end output from 'ceph osd df': > > MIN/MAX VAR: 0.73/1.31 STDDEV: 7.73 > > We have 8199 pgs total with 6775 of them in the pool that has 97% of the > data. > > The other pools are not really used (data, metadata, .rgw.root, > .rgw.control, etc). I have thought about deleting those unused pools so > that most if not all the pgs are being used by the pool with the > majority of the data. > > However...before I do that...there anything else I can do or try in > order to see if I can balance out the data more uniformly? > > Thanks in advance, > > Shain > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |