"Hello List, This could be transformed a bugreport if anyone feels like, I just kind of share my harsh experience of today. We had a few OSDs getting full while others being below 40%; while these were properly weighted (full ones of 800GB being 0.800 and fairly empty ones being 2.7TB and 2.700 weighted) it did not seem to work well. So I did a ceph osd reweight-by-utilization which resulted some stuck unclean pgs (apart from ~10% pg migration). Net wisdom said that the CRUSH map and the probabilities were the cause (of some not really defined way, mentioning probabilities rejecting OSDs, which without context were pretty hard to interpret but I have accepted "crush problem, don't ask more"), and some mentioned that CRUSH tunables should be set to optimal. I have tried to see what 'optimal' would change but that's not trivial, there seem to be no documentation to _see_ the current values [update: I figured out that exporting the crushmap and decompiling it with crushtool lists the current tunables at the top] or to know which preset contain what values. This is ceph version 0.94.9 (fe6d859066244b97b24f09d46552afc2071e6f90), aka hammer. It was installed as hammer as far as I remember. Now, the documentation says that if I set tunables to optimal, quote: "this will result in some data movement (possibly as much as 10%)." (Sidenote: ceph wasn't complaining about tunables.) So, that's okay. "ceph osd crush tunables optimal" Setting it resulted the not quite funny amount of 65% displaced objects, which didn't make me happy, nor the cluster members due to extreme IO loads. Fortunately setting it back to "legacy" caused the whole shitstorm to stop. (I will start it soon, now, in the late evening, where it won't cause too much harm.) So, it's not always "some" and "possibly as much as 10%". Reading the various tunable profiles it seems that there are changes with high data migration so I don't quite see why this "small data movement" is mentioned: it's possible, but not compulsory. Peter _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |