Hi Istvan, I have not given Octopus another try yet. But as far as I remember Manuel figured out the root cause. Maybe he can give more insights. Best, Peter Am 28.10.21 um 13:07 schrieb Szabo, Istvan (Agoda):
Hi Peter, Have you figured out what was the issue? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@xxxxxxxxx --------------------------------------------------- -----Original Message----- From: Peter Lieven <pl@xxxxxxx> Sent: Thursday, November 26, 2020 11:37 PM To: ceph-users@xxxxxxx Subject: octupus: stall i/o during recovery Email received from outside the company. If in doubt don't click links nor open attachments! ________________________________ Hi, I am currently evaluating ceph and stumbled across an odd issue when an osd comes back online. The osd was taken offline, but is still "in" and is brought back online before it is marked "out". As a test I run a fio job with 4k rand I/O on a 10G rbd volume during the OSD down and up procedure. As OSDs I use 8x 960GB SAS SSDs on 4 Nodes interconnected with 2x10GE each. Network and SSD seems not to be congested at any time. From time to time I see complete stall in the fio benchmark for approx. 10 seconds while recovery is ongoing. All recovery parameteres (max_recovery, max_backfill, sleep etc.) do not seem to influence it. While digging deeper I found that the requests are hanging in the "waiting for readable" state. Help how to debug this further would be great. Might it be linked to the new feature in Octopus to read from all OSDs and not just the primary? Thank you, Peter _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
-- Mit freundlichen Grüßen Peter Lieven ........................................................... KAMP Netzwerkdienste GmbH Vestische Str. 89-91 | 46117 Oberhausen Tel: +49 (0) 208.89 402-50 | Fax: +49 (0) 208.89 402-40 pl@xxxxxxx | http://www.kamp.de Geschäftsführer: Heiner Lante | Michael Lante Amtsgericht Duisburg | HRB Nr. 12154 USt-Id-Nr.: DE 120607556 ........................................................... _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
