Re: Clarification on sequence of recovery and client ops after OSDs rejoin cluster (also, slow requests)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Josh,

> On Sep 16, 2017, at 3:13 AM, Josh Durgin <jdurgin@xxxxxxxxxx> wrote:
> 
> (Sorry for top posting, this email client isn't great at editing)

Thanks for taking the time to respond. :)

> The mitigation strategy I mentioned before of forcing backfill could be backported to jewel, but I don't think it's a very good option for RBD users without SSDs.

Interestingly enough, we don’t see this problem on our pure SSD pool.

> In luminous there is a command (something like 'ceph pg force-recovery') that you can use to prioritize recovery of particular PGs (and thus rbd images with some scripting). This would at least let you limit the scope of affected images. A couple folks from OVH added it for just this purpose.

Uhm. I haven’t measured, but my impression is that for us it’s all over the map anyway. I don’t think we’d have many PGs that have objects of only specific rbd images … why would that happen anyway?

> Neither of these is an ideal workaround, but I haven't thought of a better one for existing versions.

I’ll discuss more strategies with Florian today, however, a few questions arise:

a) Do you have any ideas whether certain settings (recovery / backfill limits, network / disk / cpu saturation, ceph version) may be contributing in a way that this seems to hurt us more than others?

   I’m also surprised that a prioritized recovery causes 30-60 seconds of delay for a single IOP. I mean, I understand degraded throughput and latency during recovery, but what gets me are those extremely blocked individual operations.

   After we reviewed other’s settings incl. last years Cern recommendations we’ve set the following “interesting” options. Did we maybe unintentionally hit a combination that worsens this behaviour? Could the “backfill scan” and “max chunk” options make this worse?

   fd cache size = 2048
   filestore max sync interval = 60 # fsync files every 60s
   filestore op threads = 8  # more threads where needed
   filestore queue max ops = 100 # allow more queued ops
   filestore fiemap = true
   osd backfill scan max = 128
   osd backfill scan min = 32
   osd max backfills = 5
   osd recovery max active = 3
   osd recovery max single start = 1
   osd recovery op priority = 1
   osd recovery threads = 1
   osd recovery max chunk = 1048576
   osd disk threads = 1
   osd disk thread ioprio class = idle
   osd disk thread ioprio priority = 0
   osd snap trim sleep = 0.5 # throttle some long lived OSD ops
   osd op threads = 4                      # more threads where needed

   The full OSD config is here (for a week from now on):
   http://dpaste.com/35ABA0N

b) I just upgraded to hammer 0.94.10 (+segfault fix) in our development environment and _may_ have seen an improvement on this. Could this be

http://tracker.ceph.com/issues/16128

c) Are all Ceph users just silently happy with this and are we the only ones where this makes us feel uneasy? Or are we the only ones hit by this? (Well, I guess others are. Alibaba seems to have been working on the async + partial recovery, too.)

d) with size=3/min_size=2 we like to perform _quick_ maintenance operations (i.e. a simple host reboot) without evacuating the host. However, with the situation regarding recovery having such a high impact I’m now considering to do just that. Is everyone else just doing host evacuations all the time?

We’ve become edgy about this behavior as we had a couple of weeks where we were fighting against a CPU bug that caused host reboots every few days, triggering availability issues for our customer services due to slow requests.

Kind regards,
Christian

--
Christian Theune · ct@xxxxxxxxxxxxxxx · +49 345 219401 0
Flying Circus Internet Operations GmbH · http://flyingcircus.io
Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick

Attachment: signature.asc
Description: Message signed with OpenPGP

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux