It would be interesting to see if your script produces the same weights
that my script does! Would you be willing to try running the script I
posted earlier on your cluster? It does not modify anything, just
analyses the content of ceph pg dump. I want to see if it thinks your
weights are optimal for whatever pool you are targeting.
Script is here:
https://github.com/ceph/cbt/blob/master/tools/readpgdump.py
You can run it by doing ceph pg dump | readpgdump.py
Mark
On 04/14/2015 10:43 PM, GuangYang wrote:
We have a tiny script which does the CRUSH re-weight based on the PGs/OSD to achieve balance across OSDs, and we run the script right after setup the cluster to avoid data migration after the cluster is filled up.
A couple of experiences to share:
1> As suggested, it is helpful to choose a 2-powered PG number so that objects/PG is even (it is pretty even in our deployment, given the object size and disk size we have).
2> With running the script, we try to achieve even PGs/OSD (for the data pool), so that the disk utilization is most likely to be even after the cluster is filled up.
3> With disk replacement procedure (depending on the procedure you have), you may need some extra steps to make sure the CRUSH weight persist across disk replacement.
Sage has a built-in version (reweight-by-pg) for that, and here is our script - https://github.com/guangyy/ceph_misc/blob/master/osd_crush_reweight/ceph_osd_crush_reweight.pl
Hope that helps.
Thanks,
Guang
----------------------------------------
To: ceph-users@xxxxxxxxxxxxxx
From: pengyujian5201314@xxxxxxx
Date: Wed, 15 Apr 2015 01:58:08 +0000
Subject: ceph data not well distributed.
I have a ceph cluster with 125 osds with the same weight.
But I found that data is not well distributed.
df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 47929224 2066208 43405264 5% /
udev 16434372 4 16434368 1% /dev
tmpfs 6578584 728 6577856 1% /run
none 5120 0 5120 0% /run/lock
none 16446460 0 16446460 0% /run/shm
/dev/sda6 184307 62921 107767 37% /boot
/dev/mapper/osd-104 877797376 354662904 523134472 41% /ceph-osd/osd-104
/dev/mapper/osd-105 877797376 596911248 280886128 69% /ceph-osd/osd-105
/dev/mapper/osd-106 877797376 497968080 379829296 57% /ceph-osd/osd-106
/dev/mapper/osd-107 877797376 640225368 237572008 73% /ceph-osd/osd-107
/dev/mapper/osd-108 877797376 509972412 367824964 59% /ceph-osd/osd-108
/dev/mapper/osd-109 877797376 581435864 296361512 67% /ceph-osd/osd-109
/dev/mapper/osd-110 877797376 724248740 153548636 83% /ceph-osd/osd-110
/dev/mapper/osd-111 877797376 495883796 381913580 57% /ceph-osd/osd-111
/dev/mapper/osd-112 877797376 488635912 389161464 56% /ceph-osd/osd-112
/dev/mapper/osd-113 877797376 613807596 263989780 70% /ceph-osd/osd-113
/dev/mapper/osd-114 877797376 633144408 244652968 73% /ceph-osd/osd-114
/dev/mapper/osd-115 877797376 519702956 358094420 60% /ceph-osd/osd-115
/dev/mapper/osd-116 877797376 449834752 427962624 52% /ceph-osd/osd-116
/dev/mapper/osd-117 877797376 641484036 236313340 74% /ceph-osd/osd-117
/dev/mapper/osd-118 877797376 519416488 358380888 60% /ceph-osd/osd-118
/dev/mapper/osd-119 877797376 599926788 277870588 69% /ceph-osd/osd-119
/dev/mapper/osd-120 877797376 460384476 417412900 53% /ceph-osd/osd-120
/dev/mapper/osd-121 877797376 646286724 231510652 74% /ceph-osd/osd-121
/dev/mapper/osd-122 877797376 647260752 230536624 74% /ceph-osd/osd-122
/dev/mapper/osd-123 877797376 432367436 445429940 50% /ceph-osd/osd-123
/dev/mapper/osd-124 877797376 595846772 281950604 68% /ceph-osd/osd-124
The osd.104 is 41% full, but osd.110 is 83%.
Can I move some pgs from osd.110 to osd.104 manually?
Thanks!
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
