Hello everybody,
I'm very new to ceph and installed a testenvironment (nautilus).
The current goal of this cluster is, to be a short period backup.
For this goal we want to use older, mixed hardware, so I was thinking, for testing I will set up very unbalanced nodes (you can learn the most, from exceptional circumstances, right?).
I created for my cephfs two pools, one for metadata and one for storage data.
I have three nodes and the ceph osd tree looks like this:
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 21.54213 root default
-3 0.75679 host node1
0 hdd 0.75679 osd.0 up 0.00636 1.00000
-5 5.39328 host node2
1 hdd 2.66429 osd.1 up 0.65007 1.00000
3 hdd 2.72899 osd.3 up 0.65007 1.00000
-10 15.39206 host node3
5 hdd 7.27739 osd.5 up 1.00000 1.00000
6 hdd 7.27739 osd.6 up 1.00000 1.00000
2 ssd 0.38249 osd.2 up 1.00000 1.00000
4 ssd 0.45479 osd.4 up 1.00000 1.00000
The PGs and thus the data is extremely unbalanced, you can see it in the ceph osd df overview:
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
0 hdd 0.75679 0.00636 775 GiB 651 GiB 650 GiB 88 KiB 1.5 GiB 124 GiB 84.02 7.26 112 up
1 hdd 2.66429 0.65007 2.7 TiB 497 GiB 496 GiB 88 KiB 1.2 GiB 2.2 TiB 18.22 1.57 81 up
3 hdd 2.72899 0.65007 2.7 TiB 505 GiB 504 GiB 8 KiB 1.3 GiB 2.2 TiB 18.07 1.56 88 up
5 hdd 7.27739 1.00000 7.3 TiB 390 GiB 389 GiB 8 KiB 1.2 GiB 6.9 TiB 5.24 0.45 67 up
6 hdd 7.27739 1.00000 7.3 TiB 467 GiB 465 GiB 64 KiB 1.3 GiB 6.8 TiB 6.26 0.54 78 up
2 ssd 0.38249 1.00000 392 GiB 14 GiB 13 GiB 11 KiB 1024 MiB 377 GiB 3.68 0.32 2 up
4 ssd 0.45479 1.00000 466 GiB 28 GiB 27 GiB 4 KiB 1024 MiB 438 GiB 6.03 0.52 4 up
TOTAL 22 TiB 2.5 TiB 2.5 TiB 273 KiB 8.4 GiB 19 TiB 11.57
MIN/MAX VAR: 0.32/7.26 STDDEV: 6.87
To counteract this, I tried to turn on the balancer module.
The module is decreasing the reweight of the osd0 more and more, while ceph pg stat is telling me, there are more misplaced objects:
144 pgs: 144 active+clean+remapped; 853 GiB data, 2.5 TiB used, 19 TiB / 22 TiB avail; 30 MiB/s wr, 7 op/s; 242259/655140 objects misplaced (36.978%)
So my question is: is ceph supposed to do that?
Why are all those objects misplaced? Because of those 112 PGs on osd0?
Why are there 112 PGs on osd0? I did not set any pg settings except the number: 512
Thank you very much
Simon Sutter
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
