Re: How to slow down PG recovery when a failed OSD node come back?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Thanks a lot for the tips. 

I do not have the PG recovery with HDD system, and the issue only happens on SSD. I checked SSD related parameter settings, and did not find anything. Still in pursuit of what could be the root cause...

samuel 



huxiaoyu@xxxxxxxxxxxx
 
From: Frank Schilder
Date: 2021-08-26 09:01
To: huxiaoyu@xxxxxxxxxxxx; ceph-users
Subject: Re:  How to slow down PG recovery when a failed OSD node come back?
For luminous you should check the corresponding _ssd-config values for osd_recovery_sleep and osd_max_backfills.
 
However, I don't think you should see a problem with the defaults with luminous. In fact, I had good experience with making recovery even more aggressive than the defaults. You might want to look through the logs if there are other problems, for example, with peering taking very long or other OSDs being marked as down temporarily (the classic "a monitor marked me down but I'm still running"). Could be network or CPU bottlenecks.
 
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
 
________________________________________
From: huxiaoyu@xxxxxxxxxxxx <huxiaoyu@xxxxxxxxxxxx>
Sent: 25 August 2021 21:46:57
To: ceph-users
Subject:  How to slow down PG recovery when a failed OSD node come back?
 
Dear Cepher,
 
I had an all flash 3 node Ceph cluster, each node of 8 SSDs as OSDs, running Ceph release 12.2.13. I have the following setting
    osd_op_queue = wpq
    osd_op_queue_cut_off = high
and
    osd_recovery_sleep= 0.5
  osd_min_pg_log_entries = 3000
    osd_max_pg_log_entries = 10000
osd_max_backfills = 1
 
The problem i encountered is the following: After a failed OSD node come back and re-join, there is 3-5 mimutes period during which the recovery workload overwhelming the system, making user IO almost stall. After this 3-5 mimutes, the recovery process seems to calm down and slow down to a reasonable level, give priority to user IO workload.
 
What happens during the crazy 3-5 minutes? and how to reduce the negative impact then?
 
any suggestions and comments are highly appreciated,
 
best regards,
 
Samuel
 
 
 
huxiaoyu@xxxxxxxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
 
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux