Hello Istvan, as described on the ceph docu page this ratio will be multiplied with the value from osd_heartbeat_grace which is defaulted to 20 seconds. The product is 16 seconds which is the lease ticket time. So your requests can theoretically be blocked up to 16 seconds. For our usecase this is way to long to accept this while doing normal maintenance of the hosts like os-updates or something like this. with 0.2 the lease ticket times are only 4 seconds which is not realy nice but we could live with this. The smaller the value will get, the greater the cpu load will get because of renew this lease tickets. This issue we could reproduce with a 3 Node 12 OSD Cluster which only containd a coulple of test objects. Until now I didn't find a satisfying solution. Manuel On Thu, 28 Oct 2021 11:56:48 +0000 "Szabo, Istvan (Agoda)" <Istvan.Szabo@xxxxxxxxx> wrote: > Running into the same issue, have you got any answer for the side > effect? Also what's the reason you've set it to 0.2? > > I started to have this issue above 1 billions of objects and still > suffering with it. > > -----Original Message----- > From: Manuel Lausch <manuel.lausch@xxxxxxxx> > Sent: Friday, June 11, 2021 6:35 PM > To: ceph-users@xxxxxxx > Subject: [Suspicious newsletter] Re: slow ops at > restarting OSDs (octopus) > > Okay, I poked around a bit more and found this document: > https://docs.ceph.com/en/latest/dev/osd_internals/stale_read/ > > I don't understand exactly what it is all about and how it works, and > what the intetion is behind it. But there is one config option > mentiond: "osd_pool_default_read_lease_ratio" This is defaulted to > 0.8. Multiplied with the osd_hearbeat_grace (which is default 20) it > sets that "read lease" to 16 seconds ?! > > I set this ratio to 0.2 which leads to 4 seconds lease time. With > that, the problem is solved. No more slow ops. > > Until now, I thought that this is a problem on huge clusters. But > with this setting I assumed that this should be a issue with quite > small cluster as well. So I tested it with a 3 Node 12 OSD SSD > Cluster on octopus with the same issues. > > I can't believe I am the first one, which have this problem. > > > Manuel > > > On Thu, 10 Jun 2021 17:45:02 +0200 > Manuel Lausch <manuel.lausch@xxxxxxxx> wrote: > > > Hi Peter, > > > > your suggestion pointed me to the right spot. > > I didn't know about the feature, that ceph will read from replica > > PGs. > > > > So on. I found two functions in the osd/PrimaryLogPG.cc: > > "check_laggy" and "check_laggy_requeue". On both is first a check, > > if the partners have the octopus features. if not, the function is > > skipped. This explains the beginning of the problem after about the > > half cluster was updated. > > > > To verifiy this, I added "return true" in the first line of the > > functions. The issue is gone with it. But I don't know what > > problems this could trigger. I know, the root cause is not fixed > > with it. I think I will open a bug ticket with this knowlage. > > > > > > osd_op_queue_cutoff is set to high > > and a icmp rate limiting should not happen > > > > > > Thanks > > Manuel > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an > email to ceph-users-leave@xxxxxxx -- Manuel Lausch Systemadministrator Storage Services 1&1 Mail & Media Development & Technology GmbH | Brauerstraße 48 | 76135 Karlsruhe | Germany Phone: +49 721 91374-1847 E-Mail: manuel.lausch@xxxxxxxx | Web: www.1und1.de Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 5452 Geschäftsführer: Alexander Charles, Thomas Ludwig, Jan Oetjen, Sascha Vollmer Member of United Internet Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte Informationen enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat sind oder diese E-Mail irrtümlich erhalten haben, unterrichten Sie bitte den Absender und vernichten Sie diese E-Mail. Anderen als dem bestimmungsgemäßen Adressaten ist untersagt, diese E-Mail zu speichern, weiterzuleiten oder ihren Inhalt auf welche Weise auch immer zu verwenden. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient of this e-mail, you are hereby notified that saving, distribution or use of the content of this e-mail in any way is prohibited. If you have received this e-mail in error, please notify the sender and delete the e-mail. _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |