Hi Peter, Ceph cluster version is 0.94.5 and we are running with Firefly tunables and also we have 10KPGs instead of the 30K / 40K we should have. The linux kernel version is 3.10.0-327.36.1.el7.x86_64 with RHEL 7.2 On our side we havethe following settings: mon_osd_adjust_heartbeat_grace = false mon_osd_adjust_down_out_interval = false mon_osd_min_down_reporters = 5 mon_osd_min_down_reports = 10 explaining why the OSDs are not flapping but still they are behaving wrongly and generate the slow requests I am describing. The osd_op_complaint_time is with the default value (30 sec), not sure I want to change it base on your experience Thomas -----Original Message----- From: Peter Maloney [mailto:peter.maloney@xxxxxxxxxxxxxxxxxxxx] Sent: mardi 15 novembre 2016 13:44 To: Thomas Danan Cc: ceph-users@xxxxxxxxxxxxxx Subject: Re: ceph cluster having blocke requests very frequently Which kernel version are you using? I have a similar issue..ubuntu 14.04 kernel 3.13.0-96-generic, and ceph jewel 10.2.3. I get logs like this: 2016-11-15 13:13:57.295067 osd.9 10.3.0.132:6817/24137 98 : cluster [WRN] 16 slow requests, 5 included below; oldest blocked for > 7.957045 secs I set osd_op_complaint_time=5 instead of default of 30. But also had lots of ones >30 even >60s, and with even small blocks like 5s, qemu can hang, and only SIGKILL will stop it. (also I have firefly tunables set right now, and too few pgs...working on that) ________________________________ This electronic message contains information from Mycom which may be privileged or confidential. The information is intended to be for the use of the individual(s) or entity named above. If you are not the intended recipient, be aware that any disclosure, copying, distribution or any other use of the contents of this information is prohibited. If you have received this electronic message in error, please notify us by post or telephone (to the numbers or correspondence address above) or by email (at the email address above) immediately. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |