|
Hi all, We are still very new to running a Ceph cluster and have run a RGW cluster for a while now (6-ish mo), it mainly holds large DB backups (Write once, read once, delete after N days). The system is now warning us about an OSD that is near_full
and so we went to look at the usage across OSDs. We are somewhat surprised at how imbalanced the usage is across the OSDs, with the lowest usage at 22% full, the highest at nearly 90%, and an almost linear usage pattern across the OSDs (though it looks to
step in roughly 5% increments): [root@carf-ceph-osd01 ~]# ceph osd df | sort -nk8 ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS 77 hdd 7.27730 1.00000 7451G 1718G 5733G 23.06 0.43 32 73 hdd 7.27730 1.00000 7451G 1719G 5732G 23.08 0.43 31 3 hdd 7.27730 1.00000 7451G 2059G 5392G 27.63 0.52 27 46 hdd 7.27730 1.00000 7451G 2060G 5391G 27.65 0.52 32 48 hdd 7.27730 1.00000 7451G 2061G 5390G 27.66 0.52 25 127 hdd 7.27730 1.00000 7451G 2066G 5385G 27.73 0.52 31 42 hdd 7.27730 1.00000 7451G 2067G 5384G 27.74 0.52 42 107 hdd 7.27730 1.00000 7451G 2402G 5049G 32.24 0.61 34 56 hdd 7.27730 1.00000 7451G 2405G 5046G 32.28 0.61 37 51 hdd 7.27730 1.00000 7451G 2406G 5045G 32.29 0.61 30 106 hdd 7.27730 1.00000 7451G 2408G 5043G 32.31 0.61 29 81 hdd 7.27730 1.00000 7451G 2408G 5043G 32.32 0.61 25 123 hdd 7.27730 1.00000 7451G 2411G 5040G 32.37 0.61 35 47 hdd 7.27730 1.00000 7451G 2412G 5039G 32.37 0.61 29 122 hdd 7.27730 1.00000 7451G 2749G 4702G 36.90 0.69 30 84 hdd 7.27730 1.00000 7451G 2750G 4701G 36.91 0.69 35 114 hdd 7.27730 1.00000 7451G 2751G 4700G 36.92 0.69 26 82 hdd 7.27730 1.00000 7451G 2751G 4700G 36.92 0.69 43 103 hdd 7.27730 1.00000 7451G 2753G 4698G 36.94 0.69 39 36 hdd 7.27730 1.00000 7451G 2752G 4699G 36.94 0.69 37 105 hdd 7.27730 1.00000 7451G 2754G 4697G 36.97 0.69 26 14 hdd 7.27730 1.00000 7451G 3091G 4360G 41.49 0.78 31 2 hdd 7.27730 1.00000 7451G 3091G 4360G 41.49 0.78 43 8 hdd 7.27730 1.00000 7451G 3091G 4360G 41.49 0.78 37 20 hdd 7.27730 1.00000 7451G 3092G 4359G 41.50 0.78 28 60 hdd 7.27730 1.00000 7451G 3092G 4359G 41.50 0.78 29 69 hdd 7.27730 1.00000 7451G 3092G 4359G 41.50 0.78 37 110 hdd 7.27730 1.00000 7451G 3093G 4358G 41.51 0.78 38 68 hdd 7.27730 1.00000 7451G 3092G 4358G 41.51 0.78 34 76 hdd 7.27730 1.00000 7451G 3093G 4358G 41.51 0.78 28 99 hdd 7.27730 1.00000 7451G 3092G 4358G 41.51 0.78 34 50 hdd 7.27730 1.00000 7451G 3095G 4356G 41.54 0.78 35 95 hdd 7.27730 1.00000 7451G 3095G 4356G 41.54 0.78 31 0 hdd 7.27730 1.00000 7451G 3096G 4355G 41.55 0.78 36 125 hdd 7.27730 1.00000 7451G 3096G 4355G 41.55 0.78 34 128 hdd 7.27730 1.00000 7451G 3095G 4355G 41.55 0.78 37 94 hdd 7.27730 1.00000 7451G 3096G 4355G 41.55 0.78 33 63 hdd 7.27730 1.00000 7451G 3096G 4355G 41.56 0.78 41 30 hdd 7.27730 1.00000 7451G 3100G 4351G 41.60 0.78 31 26 hdd 7.27730 1.00000 7451G 3435G 4015G 46.11 0.87 30 64 hdd 7.27730 1.00000 7451G 3435G 4016G 46.11 0.87 42 57 hdd 7.27730 1.00000 7451G 3437G 4014G 46.12 0.87 29 33 hdd 7.27730 1.00000 7451G 3437G 4014G 46.13 0.87 27 65 hdd 7.27730 1.00000 7451G 3439G 4012G 46.15 0.87 29 109 hdd 7.27730 1.00000 7451G 3439G 4012G 46.16 0.87 39 11 hdd 7.27730 1.00000 7451G 3441G 4010G 46.18 0.87 32 121 hdd 7.27730 1.00000 7451G 3441G 4010G 46.18 0.87 46 78 hdd 7.27730 1.00000 7451G 3441G 4010G 46.18 0.87 36 13 hdd 7.27730 1.00000 7451G 3442G 4009G 46.19 0.87 40 115 hdd 7.27730 1.00000 7451G 3443G 4008G 46.21 0.87 33 41 hdd 7.27730 1.00000 7451G 3444G 4007G 46.22 0.87 37 49 hdd 7.27730 1.00000 7451G 3776G 3674G 50.68 0.95 34 71 hdd 7.27730 1.00000 7451G 3776G 3675G 50.68 0.95 36 97 hdd 7.27730 1.00000 7451G 3776G 3675G 50.68 0.95 26 17 hdd 7.27730 1.00000 7451G 3777G 3674G 50.70 0.95 35 75 hdd 7.27730 1.00000 7451G 3778G 3673G 50.70 0.95 41 1 hdd 7.27730 1.00000 7451G 3779G 3672G 50.71 0.95 40 79 hdd 7.27730 1.00000 7451G 3778G 3672G 50.71 0.95 42 54 hdd 7.27730 1.00000 7451G 3779G 3672G 50.72 0.95 39 58 hdd 7.27730 1.00000 7451G 3780G 3670G 50.74 0.95 41 7 hdd 7.27730 1.00000 7451G 3781G 3670G 50.74 0.95 40 21 hdd 7.27730 1.00000 7451G 3783G 3668G 50.77 0.95 27 31 hdd 7.27730 1.00000 7451G 3783G 3668G 50.77 0.95 34 67 hdd 7.27730 1.00000 7451G 3784G 3667G 50.79 0.95 33 43 hdd 7.27730 1.00000 7451G 4119G 3332G 55.28 1.04 36 72 hdd 7.27730 1.00000 7451G 4120G 3331G 55.30 1.04 45 74 hdd 7.27730 1.00000 7451G 4121G 3330G 55.31 1.04 32 102 hdd 7.27730 1.00000 7451G 4123G 3328G 55.33 1.04 35 34 hdd 7.27730 1.00000 7451G 4123G 3328G 55.33 1.04 37 111 hdd 7.27730 1.00000 7451G 4123G 3327G 55.34 1.04 40 44 hdd 7.27730 1.00000 7451G 4123G 3328G 55.34 1.04 41 27 hdd 7.27730 1.00000 7451G 4124G 3327G 55.35 1.04 44 39 hdd 7.27730 1.00000 7451G 4124G 3327G 55.35 1.04 36 55 hdd 7.27730 1.00000 7451G 4124G 3327G 55.35 1.04 45 80 hdd 7.27730 1.00000 7451G 4125G 3326G 55.36 1.04 35 116 hdd 7.27730 1.00000 7451G 4125G 3326G 55.37 1.04 47 98 hdd 7.27730 1.00000 7451G 4126G 3325G 55.38 1.04 41 132 hdd 7.27730 1.00000 7451G 4128G 3323G 55.40 1.04 43 89 hdd 7.27730 1.00000 7451G 4130G 3321G 55.43 1.04 44 6 hdd 7.27730 1.00000 7451G 4461G 2990G 59.87 1.12 32 91 hdd 7.27730 1.00000 7451G 4462G 2989G 59.88 1.12 39 124 hdd 7.27730 1.00000 7451G 4465G 2986G 59.92 1.12 30 28 hdd 7.27730 1.00000 7451G 4465G 2985G 59.93 1.12 32 92 hdd 7.27730 1.00000 7451G 4465G 2986G 59.93 1.12 41 10 hdd 7.27730 1.00000 7451G 4466G 2985G 59.94 1.13 36 25 hdd 7.27730 1.00000 7451G 4467G 2984G 59.95 1.13 35 85 hdd 7.27730 1.00000 7451G 4467G 2984G 59.95 1.13 38 12 hdd 7.27730 1.00000 7451G 4467G 2984G 59.96 1.13 46 22 hdd 7.27730 1.00000 7451G 4468G 2983G 59.96 1.13 40 40 hdd 7.27730 1.00000 7451G 4469G 2982G 59.98 1.13 43 53 hdd 7.27730 1.00000 7451G 4469G 2982G 59.98 1.13 33 88 hdd 7.27730 1.00000 7451G 4469G 2982G 59.98 1.13 36 118 hdd 7.27730 1.00000 7451G 4470G 2981G 59.99 1.13 39 86 hdd 7.27730 1.00000 7451G 4470G 2981G 59.99 1.13 40 90 hdd 7.27730 1.00000 7451G 4471G 2980G 60.01 1.13 48 100 hdd 7.27730 1.00000 7451G 4473G 2978G 60.02 1.13 34 112 hdd 7.27730 1.00000 7451G 4473G 2978G 60.03 1.13 35 24 hdd 7.27730 1.00000 7451G 4475G 2976G 60.06 1.13 36 117 hdd 7.27730 1.00000 7451G 4806G 2645G 64.49 1.21 34 66 hdd 7.27730 1.00000 7451G 4805G 2646G 64.49 1.21 37 119 hdd 7.27730 1.00000 7451G 4806G 2645G 64.50 1.21 41 93 hdd 7.27730 1.00000 7451G 4807G 2644G 64.51 1.21 34 16 hdd 7.27730 1.00000 7451G 4809G 2642G 64.54 1.21 38 101 hdd 7.27730 1.00000 7451G 4812G 2639G 64.58 1.21 36 104 hdd 7.27730 1.00000 7451G 4812G 2639G 64.58 1.21 33 15 hdd 7.27730 1.00000 7451G 4812G 2639G 64.58 1.21 39 133 hdd 7.27730 1.00000 7451G 4814G 2637G 64.61 1.21 34 4 hdd 7.27730 1.00000 7451G 4814G 2637G 64.61 1.21 38 62 hdd 7.27730 1.00000 7451G 4815G 2636G 64.62 1.21 39 9 hdd 7.27730 1.00000 7451G 4816G 2635G 64.63 1.21 46 59 hdd 7.27730 1.00000 7451G 4816G 2635G 64.64 1.21 38 38 hdd 7.27730 1.00000 7451G 4817G 2634G 64.65 1.21 42 131 hdd 7.27730 1.00000 7451G 5150G 2301G 69.12 1.30 42 32 hdd 7.27730 1.00000 7451G 5157G 2294G 69.21 1.30 42 96 hdd 7.27730 1.00000 7451G 5158G 2293G 69.22 1.30 41 83 hdd 7.27730 1.00000 7451G 5158G 2293G 69.23 1.30 40 37 hdd 7.27730 1.00000 7451G 5492G 1959G 73.70 1.38 30 108 hdd 7.27730 1.00000 7451G 5492G 1959G 73.71 1.38 35 129 hdd 7.27730 1.00000 7451G 5496G 1955G 73.75 1.38 42 18 hdd 7.27730 1.00000 7451G 5499G 1952G 73.80 1.39 37 5 hdd 7.27730 1.00000 7451G 5499G 1952G 73.80 1.39 38 130 hdd 7.27730 1.00000 7451G 5501G 1950G 73.82 1.39 41 35 hdd 7.27730 1.00000 7451G 5502G 1949G 73.83 1.39 39 70 hdd 7.27730 1.00000 7451G 5502G 1949G 73.84 1.39 46 45 hdd 7.27730 1.00000 7451G 5503G 1948G 73.86 1.39 35 126 hdd 7.27730 1.00000 7451G 5505G 1946G 73.88 1.39 42 120 hdd 7.27730 1.00000 7451G 5840G 1611G 78.37 1.47 39 23 hdd 7.27730 1.00000 7451G 5841G 1610G 78.39 1.47 40 52 hdd 7.27730 1.00000 7451G 5842G 1609G 78.40 1.47 45 61 hdd 7.27730 1.00000 7451G 5841G 1609G 78.40 1.47 41 29 hdd 7.27730 1.00000 7451G 6185G 1266G 83.01 1.56 46 87 hdd 7.27730 1.00000 7451G 6190G 1260G 83.08 1.56 43 113 hdd 7.27730 1.00000 7451G 6527G 924G 87.59 1.64 45 TOTAL 967T 515T 452T 53.27 MIN/MAX VAR: 0.43/1.64 STDDEV: 14.15 We don’t want to shoot ourselves in the foot here, so thought a quick email out to the list would be wise to get some guidance. What’s the best option to get the OSD usage rebalanced closer to even on this cluster? Is it reweighting the OSDs? Weight the bottom 25% up and the top 25% down? How do we mitigate this issue going forward? Thanks for all help in this regard! -Bryan Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
