osdmaptool target & deviation calculation

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Question about the osdmaptool deviation calculations;

For instance, 

-----
osdmaptool omap --upmap output.txt --upmap-pool cephfs_data-rep3 --upmap-max 1000 --upmap-deviation 5 
osdmaptool: osdmap file 'omap'
writing upmap command output to: output.txt
checking for upmap cleanups
upmap, max-count 1000, max deviation 5
 limiting to pools cephfs_data-rep3 ([30])
pools cephfs_data-rep3 
prepared 0/1000 changes
Unable to find further optimization, or distribution is already perfect
-----

The evaluated pool is all-on-hdd, and the pool was created with PGS > number of hdd OSDs in the cluster. So each hdd OSD is being used at least once by this pool.

Is it correct to assume that the osdmaptool is relying on the equations set at
ceph-17.2.5/src/osd/OSDMap.cc:5143

5143   // This function calculates the 2 maps osd_deviation and deviation_osd which 
5144   // hold the deviation between the current number of PGs which map to an OSD 
5145   // and the optimal number. ...

# pgs_per_weight
# ceph-17.2.5/src/osd/OSDMap.cc:4806
4806   float pgs_per_weight = total_pgs / osd_weight_total;

# target
# ceph-17.2.5/src/osd/OSDMap.cc:5156
5156     float target = osd_weight.at(oid) * pgs_per_weight;

# deviation
# ceph-17.2.5/src/osd/OSDMap.cc:5157
5157     float deviation = (float)opgs.size() - target;

And so for pgs_per_weight I calculate

ceph -f json osd df | jq '[ .nodes[] | select (.device_class == "hdd") .pgs ] | add'
divided by
ceph -f json osd df | jq '[ .nodes[] | select (.device_class == "hdd") .crush_weight ] | add'

(each hdd OSD in this cluster has identical weight)

target = osd_weight.at(oid) * pgs_per_weight

I calculate deviation for each osd

deviation = opgs.size - target
where, opgs.size = the number of PGs at an OSD. i.e. The value of $19 for each $1, in  `ceph osd df hdd | awk '{ print $1 " " $19 }'`

The result is many many OSDs with a deviation well above the upmap_max_deviation which is at default: 5

So I am wondering if I am miscalculating something, or if I'm not aware of further things the osdmaptool is considering when formulating upmap suggestions?

-Robert
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux