New to ceph / Very unbalanced cluster

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello everybody,



I'm very new to ceph and installed a testenvironment (nautilus).

The current goal of this cluster is, to be a short period backup.

For this goal we want to use older, mixed hardware, so I was thinking, for testing I will set up very unbalanced nodes (you can learn the most, from exceptional circumstances, right?).

I created for my cephfs two pools, one for metadata and one for storage data.



I have three nodes and the ceph osd tree looks like this:
ID  CLASS WEIGHT   TYPE NAME      STATUS REWEIGHT PRI-AFF
 -1          21.54213    root default
 -3            0.75679     host node1
  0   hdd  0.75679         osd.0      up  0.00636 1.00000
 -5           5.39328     host node2
  1   hdd  2.66429         osd.1      up  0.65007 1.00000
  3   hdd  2.72899         osd.3      up  0.65007 1.00000
-10        15.39206     host node3
  5   hdd  7.27739         osd.5      up  1.00000 1.00000
  6   hdd  7.27739         osd.6      up  1.00000 1.00000
  2   ssd  0.38249         osd.2      up  1.00000 1.00000
  4   ssd  0.45479         osd.4      up  1.00000 1.00000


The PGs and thus the data is extremely unbalanced, you can see it in the ceph osd df overview:
ID CLASS WEIGHT  REWEIGHT SIZE    RAW USE DATA    OMAP    META     AVAIL   %USE  VAR  PGS STATUS
 0   hdd 0.75679  0.00636 775 GiB 651 GiB 650 GiB  88 KiB     1.5 GiB 124 GiB   84.02   7.26 112     up
 1   hdd 2.66429  0.65007   2.7 TiB 497 GiB 496 GiB  88 KiB     1.2 GiB   2.2 TiB   18.22   1.57  81     up
 3   hdd 2.72899  0.65007   2.7 TiB 505 GiB 504 GiB    8 KiB     1.3 GiB   2.2 TiB   18.07   1.56  88     up
 5   hdd 7.27739  1.00000   7.3 TiB 390 GiB 389 GiB    8 KiB     1.2 GiB   6.9 TiB     5.24   0.45  67     up
 6   hdd 7.27739  1.00000   7.3 TiB 467 GiB 465 GiB  64 KiB     1.3 GiB   6.8 TiB     6.26   0.54  78     up
 2   ssd 0.38249   1.00000  392 GiB   14 GiB   13 GiB  11 KiB 1024 MiB 377 GiB     3.68   0.32   2     up
 4   ssd 0.45479   1.00000  466 GiB   28 GiB   27 GiB    4 KiB 1024 MiB 438 GiB     6.03    0.52   4     up
                    TOTAL  22 TiB 2.5 TiB 2.5 TiB 273 KiB  8.4 GiB  19 TiB 11.57
MIN/MAX VAR: 0.32/7.26  STDDEV: 6.87

To counteract this, I tried to turn on the balancer module.

The module is decreasing the reweight of the osd0 more and more, while ceph pg stat is telling me, there are more misplaced objects:

144 pgs: 144 active+clean+remapped; 853 GiB data, 2.5 TiB used, 19 TiB / 22 TiB avail; 30 MiB/s wr, 7 op/s; 242259/655140 objects misplaced (36.978%)



So my question is: is ceph supposed to do that?
Why are all those objects misplaced? Because of those 112 PGs on osd0?
Why are there 112 PGs on osd0? I did not set any pg settings except the number: 512



Thank you very much
Simon Sutter
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux