Good morning,
I have a large (1000) osd cluster running Jewel (10.2.6). It's an object store cluster, just using RGW with two EC pools of different redundancies. Tunable are optimal:
ceph osd crush show-tunables
{
"choose_local_tries": 0,
"choose_local_fallback_tries": 0,
"choose_total_tries": 50,
"chooseleaf_descend_once": 1,
"chooseleaf_vary_r": 1,
"chooseleaf_stable": 1,
"straw_calc_version": 1,
"allowed_bucket_algs": 54,
"profile": "jewel",
"optimal_tunables": 1,
"legacy_tunables": 0,
"minimum_required_version": "jewel",
"require_feature_tunables": 1,
"require_feature_tunables2": 1,
"has_v2_rules": 1,
"require_feature_tunables3": 1,
"has_v3_rules": 0,
"has_v4_buckets": 0,
"require_feature_tunables5": 1,
"has_v5_rules": 0
}
It's about 72% full and I'm starting to hit the dreaded "nearfull" warnings. My osd utilizations range from 59% to 85%. My current approach has been to use "ceph osd crush reweight" to knock a few points off the weight of any osds that are > 84% utilized. I realized I should also probably be bumping up the weights of some osds at the low end to help direct the data in the right direction, but I have not started doing that yet. It's getting a bit complicated as I'm having some I've already weighted down pop back up again, so it takes a lot of care to do it right and not screw up in a way that would move a lot of data unnecessarily, or get into a backfill_toofull situation.
FWIW, in the past on an older cluster running Hammer I believe, I had used rewight_by_utilization in this situation. That ended poorly as it lowered some of the weights so low that crush was unable to place some pgs leading me to a lengthy process of manually correcting. Also this cluster is much larger than that one was and I'm hesitant to try to shuffle so much data at once.
This is the out of ceph osd test-reweight-by-utilization:
no change
moved 0 / 278144 (0%)
avg 259.948
stddev 15.9527 -> 15.9527 (expected baseline 16.1154)
min osd.512 with 217 -> 217 pgs (0.834783 -> 0.834783 * mean)
max osd.870 with 314 -> 314 pgs (1.20794 -> 1.20794 * mean)
oload 120
max_change 0.05
max_change_osds 4
average 0.719013
overload 0.862816
So just wondering if anyone has any advice for me here, or if I should carry on as is. I would like to get overall utilization up to at least 80% before calling it full and moving on to another, as with a cluster this size, those last few percent represent quite a lot of space.
Thanks,
Aaron
CONFIDENTIALITY NOTICE
This e-mail message and any attachments are only for the use of the intended recipient and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient, any disclosure, distribution or other use of this e-mail message or attachments is prohibited. If you have received this e-mail message in error, please delete and notify the sender immediately. Thank you.
_______________________________________________
Ceph-large mailing list
Ceph-large@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-large-ceph.com
