http://xdel.ru/downloads/ceph-log/ceph-scrub-stuck.log.gz http://xdel.ru/downloads/ceph-log/cluster-w.log.gz Here, please. I have initiated a deep-scrub of osd.1 which was lead to forever-stuck I/O requests in a short time(scrub `ll do the same). Second log may be useful for proper timestamps, as seeks on the original may took a long time. Osd processes on the specific node was restarted twice - at the beginning to be sure all config options were applied and at the end to do same plus to get rid of stuck requests. On Wed, Nov 28, 2012 at 5:35 AM, Samuel Just <sam.just@xxxxxxxxxxx> wrote: > If you can reproduce it again, what we really need are the osd logs > from the acting set of a pg stuck in scrub with > debug osd = 20 > debug ms = 1 > debug filestore = 20. > > Thanks, > -Sam > > On Sun, Nov 25, 2012 at 2:08 PM, Andrey Korolyov <andrey@xxxxxxx> wrote: >> On Fri, Nov 23, 2012 at 12:35 AM, Sage Weil <sage@xxxxxxxxxxx> wrote: >>> On Thu, 22 Nov 2012, Andrey Korolyov wrote: >>>> Hi, >>>> >>>> In the recent versions Ceph introduces some unexpected behavior for >>>> the permanent connections (VM or kernel clients) - after crash >>>> recovery, I/O will hang on the next planned scrub on the following >>>> scenario: >>>> >>>> - launch a bunch of clients doing non-intensive writes, >>>> - lose one or more osd, mark them down, wait for recovery completion, >>>> - do a slow scrub, e.g. scrubbing one osd per 5m, inside bash script, >>>> or wait for ceph to do the same, >>>> - observe a raising number of pgs stuck in the active+clean+scrubbing >>>> state (they took a master role from ones which was on killed osd and >>>> almost surely they are being written in time of crash), >>>> - some time later, clients will hang hardly and ceph log introduce >>>> stuck(old) I/O requests. >>>> >>>> The only one way to return clients back without losing their I/O state >>>> is per-osd restart, which also will help to get rid of >>>> active+clean+scrubbing pgs. >>>> >>>> First of all, I`ll be happy to help to solve this problem by providing >>>> logs. >>> >>> If you can reproduce this behavior with 'debug osd = 20' and 'debug ms = >>> 1' logging on the OSD, that would be wonderful! >>> >> >> I have tested slightly different recovery flow, please see below. >> Since there is no real harm, like frozen I/O, placement groups also >> was stuck forever on the active+clean+scrubbing state, until I >> restarted all osds (end of the log): >> >> http://xdel.ru/downloads/ceph-log/recover-clients-later-than-osd.txt.gz >> >> - start the healthy cluster >> - start persistent clients >> - add an another host with pair of OSDs, let them be in the data placement >> - wait for data to rearrange >> - [22:06 timestamp] mark OSDs out or simply kill them and wait(since I >> have an 1/2 hour delay on readjust in such case, I did ``ceph osd >> out'' manually) >> - watch for data to rearrange again >> - [22:51 timestamp] when it ends, start a manual rescrub, with >> non-zero active+clean+scrubbing-state placement groups at the end of >> process which `ll stay in this state forever until something happens >> >> After that, I can restart osds one per one, if I want to get rid of >> scrubbing states immediately and then do deep-scrub(if I don`t, those >> states will return at next ceph self-scrubbing) or do per-osd >> deep-scrub, if I have a lot of time. The case I have described in the >> previous message took place when I remove osd from data placement >> which existed on the moment when client(s) have started and indeed it >> is more harmful than current one(frozen I/O leads to hanging entire >> guest, for example). Since testing those flow took a lot of time, I`ll >> send logs related to this case tomorrow. >> >>>> Second question is not directly related to this problem, but I >>>> have thought on for a long time - is there a planned features to >>>> control scrub process more precisely, e.g. pg scrub rate or scheduled >>>> scrub, instead of current set of timeouts which of course not very >>>> predictable on when to run? >>> >>> Not yet. I would be interested in hearing what kind of control/config >>> options/whatever you (and others) would like to see! >> >> Of course it will be awesome to have any determined scheduler or at >> least an option to disable automated scrubbing, since it is not very >> determined in time and deep-scrub eating a lot of I/O if command >> issued against entire OSD. Rate limiting is not in the first place, at >> least it may be recreated in external script, but for those who prefer >> to leave control to Ceph, it may be very useful. >> >> Thanks! >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
