When a single pg is a substantial percentage of an OSD (eg 10%) it's hard for the upmap balancer to do much. It's possible you'd have just as much space, or more, if you removed the 500GB HDDs. Another option might be to mdraid-0 the 500GB's in pairs, and make OSDs from the pairs; this would result in 1TB OSDs that might work better. Any cluster on this hardware is going to be duct tape and string, though. On Thu, Mar 30, 2023 at 10:35 AM Nicola Mori <mori@xxxxxxxxxx> wrote: > > Dear Ceph users, > > my cluster is made up of 10 old machines, with uneven number of disks and disk size. Essentially I have just one big data pool (6+2 erasure code, with host failure domain) for which I am currently experiencing a very poor available space (88 TB of which 40 TB occupied, as reported by df -h on hosts mounting the cephfs) compared to the raw one (196.5 TB). I have a total of 104 OSDs and 512 PGs for the pool; I cannot increment the PG number since the machines are old and with very low amount of RAM, and some of them are already overloaded. > > In this situation I'm seeing a high occupation of small OSDs (500 MB) with respect to bigger ones (2 and 4 TB) even if the weight is set equal to disk capacity (see below for ceph osd tree). For example OSD 9 is at 62% occupancy even with weight 0.5 and reweight 0.75, while the highest occupancy for 2 TB OSDs is 41% (OSD 18) and 4 TB OSDs is 23% (OSD 79). I guess this high occupancy for 500 MB OSDs combined with erasure code size and host failure domain might be the cause of the poor available space, could this be true? The upmap balancer is currently running but I don't know if and how much it could improve the situation. > Any hint is greatly appreciated, thanks. > > Nicola > > # ceph osd tree > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > -1 196.47754 root default > -7 14.55518 host aka > 4 hdd 1.81940 osd.4 up 1.00000 1.00000 > 11 hdd 1.81940 osd.11 up 1.00000 1.00000 > 18 hdd 1.81940 osd.18 up 1.00000 1.00000 > 26 hdd 1.81940 osd.26 up 1.00000 1.00000 > 32 hdd 1.81940 osd.32 up 1.00000 1.00000 > 41 hdd 1.81940 osd.41 up 1.00000 1.00000 > 48 hdd 1.81940 osd.48 up 1.00000 1.00000 > 55 hdd 1.81940 osd.55 up 1.00000 1.00000 > -3 14.55518 host balin > 0 hdd 1.81940 osd.0 up 1.00000 1.00000 > 8 hdd 1.81940 osd.8 up 1.00000 1.00000 > 15 hdd 1.81940 osd.15 up 1.00000 1.00000 > 22 hdd 1.81940 osd.22 up 1.00000 1.00000 > 29 hdd 1.81940 osd.29 up 1.00000 1.00000 > 34 hdd 1.81940 osd.34 up 1.00000 1.00000 > 43 hdd 1.81940 osd.43 up 1.00000 1.00000 > 49 hdd 1.81940 osd.49 up 1.00000 1.00000 > -13 29.10950 host bifur > 3 hdd 3.63869 osd.3 up 1.00000 1.00000 > 14 hdd 3.63869 osd.14 up 1.00000 1.00000 > 27 hdd 3.63869 osd.27 up 1.00000 1.00000 > 37 hdd 3.63869 osd.37 up 1.00000 1.00000 > 50 hdd 3.63869 osd.50 up 1.00000 1.00000 > 59 hdd 3.63869 osd.59 up 1.00000 1.00000 > 64 hdd 3.63869 osd.64 up 1.00000 1.00000 > 69 hdd 3.63869 osd.69 up 1.00000 1.00000 > -17 29.10950 host bofur > 2 hdd 3.63869 osd.2 up 1.00000 1.00000 > 21 hdd 3.63869 osd.21 up 1.00000 1.00000 > 39 hdd 3.63869 osd.39 up 1.00000 1.00000 > 57 hdd 3.63869 osd.57 up 1.00000 1.00000 > 66 hdd 3.63869 osd.66 up 1.00000 1.00000 > 72 hdd 3.63869 osd.72 up 1.00000 1.00000 > 76 hdd 3.63869 osd.76 up 1.00000 1.00000 > 79 hdd 3.63869 osd.79 up 1.00000 1.00000 > -21 29.10376 host dwalin > 88 hdd 1.81898 osd.88 up 1.00000 1.00000 > 89 hdd 1.81898 osd.89 up 1.00000 1.00000 > 90 hdd 1.81898 osd.90 up 1.00000 1.00000 > 91 hdd 1.81898 osd.91 up 1.00000 1.00000 > 92 hdd 1.81898 osd.92 up 1.00000 1.00000 > 93 hdd 1.81898 osd.93 up 1.00000 1.00000 > 94 hdd 1.81898 osd.94 up 1.00000 1.00000 > 95 hdd 1.81898 osd.95 up 1.00000 1.00000 > 96 hdd 1.81898 osd.96 up 1.00000 1.00000 > 97 hdd 1.81898 osd.97 up 1.00000 1.00000 > 98 hdd 1.81898 osd.98 up 1.00000 1.00000 > 99 hdd 1.81898 osd.99 up 1.00000 1.00000 > 100 hdd 1.81898 osd.100 up 1.00000 1.00000 > 101 hdd 1.81898 osd.101 up 1.00000 1.00000 > 102 hdd 1.81898 osd.102 up 1.00000 1.00000 > 103 hdd 1.81898 osd.103 up 1.00000 1.00000 > -9 14.55518 host ogion > 7 hdd 1.81940 osd.7 up 1.00000 1.00000 > 16 hdd 1.81940 osd.16 up 1.00000 1.00000 > 23 hdd 1.81940 osd.23 up 1.00000 1.00000 > 33 hdd 1.81940 osd.33 up 1.00000 1.00000 > 40 hdd 1.81940 osd.40 up 1.00000 1.00000 > 47 hdd 1.81940 osd.47 up 1.00000 1.00000 > 54 hdd 1.81940 osd.54 up 1.00000 1.00000 > 61 hdd 1.81940 osd.61 up 1.00000 1.00000 > -19 14.55518 host prestno > 81 hdd 1.81940 osd.81 up 1.00000 1.00000 > 82 hdd 1.81940 osd.82 up 1.00000 1.00000 > 83 hdd 1.81940 osd.83 up 1.00000 1.00000 > 84 hdd 1.81940 osd.84 up 1.00000 1.00000 > 85 hdd 1.81940 osd.85 up 1.00000 1.00000 > 86 hdd 1.81940 osd.86 up 1.00000 1.00000 > 87 hdd 1.81940 osd.87 up 1.00000 1.00000 > 104 hdd 1.81940 osd.104 up 1.00000 1.00000 > -15 29.10376 host remolo > 6 hdd 1.81897 osd.6 up 1.00000 1.00000 > 12 hdd 1.81897 osd.12 up 1.00000 1.00000 > 19 hdd 1.81897 osd.19 up 1.00000 1.00000 > 28 hdd 1.81897 osd.28 up 1.00000 1.00000 > 35 hdd 1.81897 osd.35 up 1.00000 1.00000 > 44 hdd 1.81897 osd.44 up 1.00000 1.00000 > 52 hdd 1.81897 osd.52 up 1.00000 1.00000 > 58 hdd 1.81897 osd.58 up 1.00000 1.00000 > 63 hdd 1.81897 osd.63 up 1.00000 1.00000 > 67 hdd 1.81897 osd.67 up 1.00000 1.00000 > 71 hdd 1.81897 osd.71 up 1.00000 1.00000 > 73 hdd 1.81897 osd.73 up 1.00000 1.00000 > 74 hdd 1.81897 osd.74 up 1.00000 1.00000 > 75 hdd 1.81897 osd.75 up 1.00000 1.00000 > 77 hdd 1.81897 osd.77 up 1.00000 1.00000 > 78 hdd 1.81897 osd.78 up 1.00000 1.00000 > -5 14.55518 host rokanan > 1 hdd 1.81940 osd.1 up 1.00000 1.00000 > 10 hdd 1.81940 osd.10 up 1.00000 1.00000 > 17 hdd 1.81940 osd.17 up 1.00000 1.00000 > 24 hdd 1.81940 osd.24 up 1.00000 1.00000 > 31 hdd 1.81940 osd.31 up 1.00000 1.00000 > 38 hdd 1.81940 osd.38 up 1.00000 1.00000 > 46 hdd 1.81940 osd.46 up 1.00000 1.00000 > 53 hdd 1.81940 osd.53 up 1.00000 1.00000 > -11 7.27515 host romolo > 5 hdd 0.45470 osd.5 up 1.00000 1.00000 > 9 hdd 0.45470 osd.9 up 0.75000 1.00000 > 13 hdd 0.45470 osd.13 up 1.00000 1.00000 > 20 hdd 0.45470 osd.20 up 0.95000 1.00000 > 25 hdd 0.45470 osd.25 up 0.75000 1.00000 > 30 hdd 0.45470 osd.30 up 1.00000 1.00000 > 36 hdd 0.45470 osd.36 up 1.00000 1.00000 > 42 hdd 0.45470 osd.42 up 1.00000 1.00000 > 45 hdd 0.45470 osd.45 up 0.85004 1.00000 > 51 hdd 0.45470 osd.51 up 0.89999 1.00000 > 56 hdd 0.45470 osd.56 up 1.00000 1.00000 > 60 hdd 0.45470 osd.60 up 1.00000 1.00000 > 62 hdd 0.45470 osd.62 up 1.00000 1.00000 > 65 hdd 0.45470 osd.65 up 0.85004 1.00000 > 68 hdd 0.45470 osd.68 up 1.00000 1.00000 > 70 hdd 0.45470 osd.70 up 1.00000 1.00000 > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |