Question about the osdmaptool deviation calculations;
For instance,
-----
osdmaptool omap --upmap output.txt --upmap-pool cephfs_data-rep3 --upmap-max 1000 --upmap-deviation 5
osdmaptool: osdmap file 'omap'
writing upmap command output to: output.txt
checking for upmap cleanups
upmap, max-count 1000, max deviation 5
limiting to pools cephfs_data-rep3 ([30])
pools cephfs_data-rep3
prepared 0/1000 changes
Unable to find further optimization, or distribution is already perfect
-----
The evaluated pool is all-on-hdd, and the pool was created with PGS > number of hdd OSDs in the cluster. So each hdd OSD is being used at least once by this pool.
Is it correct to assume that the osdmaptool is relying on the equations set at
ceph-17.2.5/src/osd/OSDMap.cc:5143
5143 // This function calculates the 2 maps osd_deviation and deviation_osd which
5144 // hold the deviation between the current number of PGs which map to an OSD
5145 // and the optimal number. ...
# pgs_per_weight
# ceph-17.2.5/src/osd/OSDMap.cc:4806
4806 float pgs_per_weight = total_pgs / osd_weight_total;
# target
# ceph-17.2.5/src/osd/OSDMap.cc:5156
5156 float target = osd_weight.at(oid) * pgs_per_weight;
# deviation
# ceph-17.2.5/src/osd/OSDMap.cc:5157
5157 float deviation = (float)opgs.size() - target;
And so for pgs_per_weight I calculate
ceph -f json osd df | jq '[ .nodes[] | select (.device_class == "hdd") .pgs ] | add'
divided by
ceph -f json osd df | jq '[ .nodes[] | select (.device_class == "hdd") .crush_weight ] | add'
(each hdd OSD in this cluster has identical weight)
target = osd_weight.at(oid) * pgs_per_weight
I calculate deviation for each osd
deviation = opgs.size - target
where, opgs.size = the number of PGs at an OSD. i.e. The value of $19 for each $1, in `ceph osd df hdd | awk '{ print $1 " " $19 }'`
The result is many many OSDs with a deviation well above the upmap_max_deviation which is at default: 5
So I am wondering if I am miscalculating something, or if I'm not aware of further things the osdmaptool is considering when formulating upmap suggestions?
-Robert
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
