Hi, Many of my osds having this issue which causes 10-15ms osd write operation latency and more than 60ms read operation latency. This causes rgw wait for operations and after a while rgw just restarted (all of them in my cluster) and only available after slow ops disappeared. I see similar issue but haven't really seen solution anywhere: https://tracker.ceph.com/issues/44184 I'm facing this issue in 2 of my cluster's from my 3 clusters multisite environment (octopus 15.2.14). Some background information, where I'm facing this issues, before I had many flapping osds even some unfound objects, not sure would that be related to this. 2021-10-12T09:59:45.542+0700 7fa0445a7700 -1 osd.46 32739 get_health_metrics reporting 205 slow ops, oldest is osd_op(client.115442393.0:1420913395 28.23s0 28:c4b40264:::9213182a-14ba-48ad-bde9-289a1c0c0de8.29868038.12_geo%2fpoi%2f1718955%2f7fc1308d421939a23614908dda8ff659.jpg:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e32739) 2021-10-12T09:59:46.583+0700 7fa0445a7700 -1 osd.46 32739 get_health_metrics reporting 205 slow ops, oldest is osd_op(client.115442393.0:1420913395 28.23s0 28:c4b40264:::9213182a-14ba-48ad-bde9-289a1c0c0de8.29868038.12_geo%2fpoi%2f1718955%2f7fc1308d421939a23614908dda8ff659.jpg:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e32739) 2021-10-12T09:59:47.581+0700 7fa0445a7700 -1 osd.46 32739 get_health_metrics reporting 205 slow ops, oldest is osd_op(client.115442393.0:1420913395 28.23s0 28:c4b40264:::9213182a-14ba-48ad-bde9-289a1c0c0de8.29868038.12_geo%2fpoi%2f1718955%2f7fc1308d421939a23614908dda8ff659.jpg:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e32739) 2021-10-12T09:59:48.551+0700 7fa0445a7700 -1 osd.46 32739 get_health_metrics reporting 205 slow ops, oldest is osd_op(client.115442393.0:1420913395 28.23s0 28:c4b40264:::9213182a-14ba-48ad-bde9-289a1c0c0de8.29868038.12_geo%2fpoi%2f1718955%2f7fc1308d421939a23614908dda8ff659.jpg:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e32739) 2021-10-12T09:59:49.592+0700 7fa0445a7700 -1 osd.46 32739 get_health_metrics reporting 205 slow ops, oldest is osd_op(client.115442393.0:1420913395 28.23s0 28:c4b40264:::9213182a-14ba-48ad-bde9-289a1c0c0de8.29868038.12_geo%2fpoi%2f1718955%2f7fc1308d421939a23614908dda8ff659.jpg:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e32739) Haven't really fund anybody in the maillist also about this :/ Thank you _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |