Thanks a lot for the tips.
I do not have the PG recovery with HDD system, and the issue only happens on SSD. I checked SSD related parameter settings, and did not find anything. Still in pursuit of what could be the root cause...
samuel
huxiaoyu@xxxxxxxxxxxx
From: Frank Schilder
Date: 2021-08-26 09:01
To: huxiaoyu@xxxxxxxxxxxx; ceph-users
Subject: Re: How to slow down PG recovery when a failed OSD node come back?
For luminous you should check the corresponding _ssd-config values for osd_recovery_sleep and osd_max_backfills.
However, I don't think you should see a problem with the defaults with luminous. In fact, I had good experience with making recovery even more aggressive than the defaults. You might want to look through the logs if there are other problems, for example, with peering taking very long or other OSDs being marked as down temporarily (the classic "a monitor marked me down but I'm still running"). Could be network or CPU bottlenecks.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: huxiaoyu@xxxxxxxxxxxx <huxiaoyu@xxxxxxxxxxxx>
Sent: 25 August 2021 21:46:57
To: ceph-users
Subject: How to slow down PG recovery when a failed OSD node come back?
Dear Cepher,
I had an all flash 3 node Ceph cluster, each node of 8 SSDs as OSDs, running Ceph release 12.2.13. I have the following setting
osd_op_queue = wpq
osd_op_queue_cut_off = high
and
osd_recovery_sleep= 0.5
osd_min_pg_log_entries = 3000
osd_max_pg_log_entries = 10000
osd_max_backfills = 1
The problem i encountered is the following: After a failed OSD node come back and re-join, there is 3-5 mimutes period during which the recovery workload overwhelming the system, making user IO almost stall. After this 3-5 mimutes, the recovery process seems to calm down and slow down to a reasonable level, give priority to user IO workload.
What happens during the crazy 3-5 minutes? and how to reduce the negative impact then?
any suggestions and comments are highly appreciated,
best regards,
Samuel
huxiaoyu@xxxxxxxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
