Hi Alex, I am searching for a log line that points me in the right direction. From what I've seen, I could find a specific Host, OSD, PG that was leading to this problem. But maybe I am looking at the wrong logs. I have around 150k lines that look like this: ceph.log.timeframe:2022-12-02T18:19:59.877920+0100 osd.122 (osd.122) 5195 : cluster [WRN] 14 slow requests (by type [ 'delayed' : 2 'waiting for sub ops' : 12 ] most affected pool [ 'rbd' : 14 ]) ceph.log.timeframe:2022-12-02T18:19:59.905505+0100 osd.118 (osd.118) 21011 : cluster [WRN] 256 slow requests (by type [ 'delayed' : 243 'waiting for sub ops' : 13 ] most affected pool [ 'rbd' : 256 ]) ceph.log.timeframe:2022-12-02T18:19:59.928599+0100 osd.120 (osd.120) 19800 : cluster [WRN] 71 slow requests (by type [ 'delayed' : 15 'waiting for sub ops' : 56 ] most affected pool [ 'rbd' : 71 ]) ceph.log.timeframe:2022-12-02T18:19:59.968535+0100 osd.54 (osd.54) 6960 : cluster [WRN] 38 slow requests (by type [ 'delayed' : 21 'waiting for sub ops' : 17 ] most affected pool [ 'rbd' : 38 ]) ceph.log.timeframe:2022-12-02T18:19:59.973174+0100 osd.97 (osd.97) 16792 : cluster [WRN] 19 slow requests (by type [ 'delayed' : 11 'waiting for sub ops' : 8 ] most affected pool [ 'rbd' : 19 ]) ceph.log.timeframe:2022-12-02T18:19:59.978565+0100 osd.42 (osd.42) 5724 : cluster [WRN] 12 slow requests (by type [ 'delayed' : 5 'waiting for sub ops' : 7 ] most affected pool [ 'rbd' : 12 ]) ceph.log.timeframe:2022-12-02T18:19:59.980684+0100 osd.98 (osd.98) 18471 : cluster [WRN] 35 slow requests (by type [ 'delayed' : 3 'waiting for sub ops' : 32 ] most affected pool [ 'rbd' : 35 ]) ceph.log.timeframe:2022-12-02T18:19:59.992514+0100 osd.77 (osd.77) 11319 : cluster [WRN] 256 slow requests (by type [ 'delayed' : 232 'waiting for sub ops' : 24 ] most affected pool [ 'rbd' : 256 ]) and around 50k that look like this: ceph-osd.99.log.timeframe:2022-12-02T18:19:59.605+0100 7ff8f96ba700 -1 osd.99 945870 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.171194478.0:4862294 8.cf5 8:af34e5b1:::rbd_header.47d6a06b8b4567:head [watch ping cookie 18446462598732840961 gen 26] snapc 0=[] ondisk+write+known_if_redirected e945870) ceph-osd.92.log.timeframe:2022-12-02T18:14:57.415+0100 7f9e8e4fd700 -1 osd.92 945870 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.177840485.0:141305 8.159f 8:f9adda1f:::rbd_data.82f60d356b4e4a.000000000000a1c2:head [write 1900544~147456 in=147456b] snapc 0=[] ondisk+write+known_if_redirected e945868) Cheers Boris Am So., 4. Dez. 2022 um 03:15 Uhr schrieb Alex Gorbachev < ag@xxxxxxxxxxxxxxxxxxx>: > Boris, I have seen one problematic OSD cause this issue on all OSD with > which its PGs peered. The solution was to take out the slow OSD, > immediately all slow ops stopped. I found it by observing common OSDs in > reported slow ops. Not saying this is your issue, but it may be a > possibility. Good luck! > > -- > Alex Gorbachev > https://alextelescope.blogspot.com > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |