Re: Openstack VM IOPS drops dramatically during Ceph recovery

["huxiaoyu@xxxxxxxxxxxx" <huxiaoyu@xxxxxxxxxxxx>]

hello, Robert 

thanks for the quick reply. I did test with  osd op queue = wpq , and     osd op queue cut off = high

and 

osd_recovery_op_priority = 1  

osd recovery delay start = 20  

osd recovery max active = 1  

osd recovery max chunk = 1048576  

osd recovery sleep = 1   

osd recovery sleep hdd = 1

osd recovery sleep ssd = 1

osd recovery sleep hybrid = 1 

osd recovery priority = 1

osd max backfills = 1     

osd backfill scan max = 16  

osd backfill scan min = 4   

osd_op_thread_suicide_timeout = 300   

But still the ceph cluster showed extremely hug recovery activities during the beginning of the recovery, and after ca. 5-10 minutes, the recovery gradually get under the control. I guess this is quite similar to what you encountered in Nov. 2015.

It is really annoying, and what else can i do to mitigate this weird inital-recovery issue? any suggestions are much appreciated.

thanks again,

samuel

---

huxiaoyu@xxxxxxxxxxxx

 

From: Robert LeBlanc

Date: 2019-10-17 21:23

To: huxiaoyu@xxxxxxxxxxxx

CC: ceph-users

Subject: Re: Re: Openstack VM IOPS drops dramatically during Ceph recovery

On Thu, Oct 17, 2019 at 12:08 PM huxiaoyu@xxxxxxxxxxxx

<huxiaoyu@xxxxxxxxxxxx> wrote:

>

> I happened to find a note that you wrote in Nov 2015: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-November/006173.html

> and I believe this is what i just hit exactly the same behavior : a host down will badly take the client performance down 1/10 (with 200MB/s recovery workload) and then took ten minutes  to get good control of OSD recovery.

>

> Could you please share how did you eventally solve that issue? by seting a fair large OSD recovery delay start or any other parameter?

 

Wow! Dusting off the cobwebs here. I think this is what lead me to dig

into the code and write the WPQ scheduler. I can't remember doing

anything specific. I'm sorry I'm not much help in this regard.

----------------

Robert LeBlanc

PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1

 

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com