Re: rebalancing ceph cluster

[Robert LeBlanc <robert@xxxxxxxxxxxxx>]

The placement of PGs is random in the cluster and takes into account any CRUSH rules which may also skew the distribution. Having more PGs will help give more options for placing PGs, but it still may not be adequate. It is recommended to have between 100-150 PGs per OSD, and you are pretty close. If you aren't planning to add any more pools, then splitting the PGs for pools that have a lot of data can help.

To get things to be more balanced, you can reweight the high utlization OSDs down to cause CRUSH to migrate some PGs off. This won't mean that they will get moved to the lowest utilized OSDs (they might wind up on another one that is pretty full). So, it may take several iterations to get things balanced. Just be sure that if you reweighted one down and it is now much lower usage than the others to reweight it back up to attract some PGs back to it.

```ceph osd reweight {osd-num} {weight}```

----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1

On Mon, Jun 24, 2019 at 2:25 AM jinguk.kwon@xxxxxxxxxxx <jinguk.kwon@xxxxxxxxxxx> wrote:

Hello everyone,

We have some osd on the ceph.

Some osd's usage is more than 77% and another osd's usage is 39% in the same host.

I wonder why osd’s usage is different.(Difference is large) and how can i fix it?

ID  CLASS   WEIGHT    REWEIGHT SIZE    USE     AVAIL   %USE  VAR  PGS TYPE NAME

 -2          93.26010        - 93.3TiB 52.3TiB 41.0TiB 56.04 0.98   -     host serverA

…...

 33 HDD  9.09511  1.00000 9.10TiB 3.55TiB 5.54TiB 39.08 0.68  66         osd.4

 45 HDD   7.27675  1.00000 7.28TiB 5.64TiB 1.64TiB 77.53 1.36  81         osd.7

…... 

-5          79.99017        - 80.0TiB 47.7TiB 32.3TiB 59.62 1.04   -     host serverB

  1 HDD   9.09511  1.00000 9.10TiB 4.79TiB 4.31TiB 52.63 0.92  87         osd.1

  6 HDD   9.09511  1.00000 9.10TiB 6.62TiB 2.48TiB 72.75 1.27  99         osd.6

 …...

Thank you

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com