307 is on ceph03.
Jeff
On Mon, Mar 7, 2016 at 6:48 PM, Samuel Just <sjust@xxxxxxxxxx> wrote:
Which node is osd.307 on?
-Sam
On Mon, Mar 7, 2016 at 4:43 PM, Samuel Just <sjust@xxxxxxxxxx> wrote:
> ' I didn't see the errors in the tracker on the new nodes, but they
> were only receiving new data, not migrating it.' -- What do you mean
> by that?
> -Sam
>
> On Mon, Mar 7, 2016 at 4:42 PM, Jeffrey McDonald <jmcdonal@xxxxxxx> wrote:
>> The filesystem is xfs everywhere, there are nine hosts. The two new ceph
>> nodes 08, 09 have a new kernel. I didn't see the errors in the tracker on
>> the new nodes, but they were only receiving new data, not migrating it.
>> Jeff
>>
>> ceph2: Linux ceph2 3.13.0-65-generic #106-Ubuntu SMP Fri Oct 2 22:08:27 UTC
>> 2015 x86_64 x86_64 x86_64 GNU/Linux
>> ceph1: Linux ceph1 3.13.0-65-generic #106-Ubuntu SMP Fri Oct 2 22:08:27 UTC
>> 2015 x86_64 x86_64 x86_64 GNU/Linux
>> ceph3: Linux ceph3 3.13.0-65-generic #106-Ubuntu SMP Fri Oct 2 22:08:27 UTC
>> 2015 x86_64 x86_64 x86_64 GNU/Linux
>> ceph03: Linux ceph03 3.13.0-65-generic #106-Ubuntu SMP Fri Oct 2 22:08:27
>> UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>> ceph01: Linux ceph01 3.13.0-65-generic #106-Ubuntu SMP Fri Oct 2 22:08:27
>> UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>> ceph02: Linux ceph02 3.13.0-65-generic #106-Ubuntu SMP Fri Oct 2 22:08:27
>> UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>> ceph06: Linux ceph06 3.13.0-65-generic #106-Ubuntu SMP Fri Oct 2 22:08:27
>> UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>> ceph05: Linux ceph05 3.13.0-65-generic #106-Ubuntu SMP Fri Oct 2 22:08:27
>> UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>> ceph04: Linux ceph04 3.13.0-65-generic #106-Ubuntu SMP Fri Oct 2 22:08:27
>> UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>> ceph08: Linux ceph08 3.19.0-51-generic #58~14.04.1-Ubuntu SMP Fri Feb 26
>> 22:02:58 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
>> ceph07: Linux ceph07 3.13.0-65-generic #106-Ubuntu SMP Fri Oct 2 22:08:27
>> UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>> ceph09: Linux ceph09 3.19.0-51-generic #58~14.04.1-Ubuntu SMP Fri Feb 26
>> 22:02:58 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
>>
>>
>> On Mon, Mar 7, 2016 at 6:39 PM, Samuel Just <sjust@xxxxxxxxxx> wrote:
>>>
>>> What filesystem and kernel are you running on the osds? This (and
>>> your other bug, actually) could be explained by some kind of weird
>>> kernel readdir behavior.
>>> -Sam
>>>
>>> On Mon, Mar 7, 2016 at 4:36 PM, Samuel Just <sjust@xxxxxxxxxx> wrote:
>>> > Hmm, so much for that theory, still looking. If you can produce
>>> > another set of logs (as before) from scrubbing that pg, it might help.
>>> > -Sam
>>> >
>>> > On Mon, Mar 7, 2016 at 4:34 PM, Jeffrey McDonald <jmcdonal@xxxxxxx>
>>> > wrote:
>>> >> they're all the same.....see attached.
>>> >>
>>> >> On Mon, Mar 7, 2016 at 6:31 PM, Samuel Just <sjust@xxxxxxxxxx> wrote:
>>> >>>
>>> >>> Have you confirmed the versions?
>>> >>> -Sam
>>> >>>
>>> >>> On Mon, Mar 7, 2016 at 4:29 PM, Jeffrey McDonald <jmcdonal@xxxxxxx>
>>> >>> wrote:
>>> >>> > I have one other very strange event happening, I've opened a ticket
>>> >>> > on
>>> >>> > it:
>>> >>> > http://tracker.ceph.com/issues/14766
>>> >>> >
>>> >>> > During this migration, OSD failed probably over 400 times while
>>> >>> > moving
>>> >>> > data
>>> >>> > around. We move the empty directories and restarted the OSDs. I
>>> >>> > can't
>>> >>> > say if this is related--I have no reason to suspect it is.
>>> >>> >
>>> >>> > Jeff
>>> >>> >
>>> >>> > On Mon, Mar 7, 2016 at 5:31 PM, Shinobu Kinjo <shinobu.kj@xxxxxxxxx>
>>> >>> > wrote:
>>> >>> >>
>>> >>> >> What could cause this kind of unexpected behaviour?
>>> >>> >> Any assumption??
>>> >>> >> Sorry for interrupting you.
>>> >>> >>
>>> >>> >> Cheers,
>>> >>> >> S
>>> >>> >>
>>> >>> >> On Tue, Mar 8, 2016 at 8:19 AM, Samuel Just <sjust@xxxxxxxxxx>
>>> >>> >> wrote:
>>> >>> >> > Hmm, at the end of the log, the pg is still inconsistent. Can
>>> >>> >> > you
>>> >>> >> > attach a ceph pg query on that pg?
>>> >>> >> > -Sam
>>> >>> >> >
>>> >>> >> > On Mon, Mar 7, 2016 at 3:05 PM, Samuel Just <sjust@xxxxxxxxxx>
>>> >>> >> > wrote:
>>> >>> >> >> If so, that strongly suggests that the pg was actually never
>>> >>> >> >> inconsistent in the first place and that the bug is in scrub
>>> >>> >> >> itself
>>> >>> >> >> presumably getting confused about an object during a write. The
>>> >>> >> >> next
>>> >>> >> >> step would be to get logs like the above from a pg as it scrubs
>>> >>> >> >> transitioning from clean to inconsistent. If it's really a race
>>> >>> >> >> between scrub and a write, it's probably just non-deterministic,
>>> >>> >> >> you
>>> >>> >> >> could set logging on a set of osds and continuously scrub any
>>> >>> >> >> pgs
>>> >>> >> >> which only map to those osds until you reproduce the problem.
>>> >>> >> >> -Sam
>>> >>> >> >>
>>> >>> >> >> On Mon, Mar 7, 2016 at 2:44 PM, Samuel Just <sjust@xxxxxxxxxx>
>>> >>> >> >> wrote:
>>> >>> >> >>> So after the scrub, it came up clean? The inconsistent/missing
>>> >>> >> >>> objects reappeared?
>>> >>> >> >>> -Sam
>>> >>> >> >>>
>>> >>> >> >>> On Mon, Mar 7, 2016 at 2:33 PM, Jeffrey McDonald
>>> >>> >> >>> <jmcdonal@xxxxxxx>
>>> >>> >> >>> wrote:
>>> >>> >> >>>> Hi Sam,
>>> >>> >> >>>>
>>> >>> >> >>>> I've done as you requested:
>>> >>> >> >>>>
>>> >>> >> >>>> pg 70.459 is active+clean+inconsistent, acting
>>> >>> >> >>>> [307,210,273,191,132,450]
>>> >>> >> >>>>
>>> >>> >> >>>> # for i in 307 210 273 191 132 450 ; do
>>> >>> >> >>>>> ceph tell osd.$i injectargs '--debug-osd 20
>>> >>> >> >>>>> --debug-filestore 20
>>> >>> >> >>>>> --debug-ms 1'
>>> >>> >> >>>>> done
>>> >>> >> >>>> debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1
>>> >>> >> >>>> debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1
>>> >>> >> >>>> debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1
>>> >>> >> >>>> debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1
>>> >>> >> >>>> debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1
>>> >>> >> >>>> debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1
>>> >>> >> >>>>
>>> >>> >> >>>>
>>> >>> >> >>>> # date
>>> >>> >> >>>> Mon Mar 7 16:03:38 CST 2016
>>> >>> >> >>>>
>>> >>> >> >>>>
>>> >>> >> >>>> # ceph pg deep-scrub 70.459
>>> >>> >> >>>> instructing pg 70.459 on osd.307 to deep-scrub
>>> >>> >> >>>>
>>> >>> >> >>>>
>>> >>> >> >>>>
>>> >>> >> >>>> Scrub finished around
>>> >>> >> >>>>
>>> >>> >> >>>> # date
>>> >>> >> >>>> Mon Mar 7 16:13:03 CST 2016
>>> >>> >> >>>>
>>> >>> >> >>>>
>>> >>> >> >>>>
>>> >>> >> >>>>
>>> >>> >> >>>> I've tar'd+gziped the files which can be downloaded from here.
>>> >>> >> >>>> The
>>> >>> >> >>>> logs
>>> >>> >> >>>> start a minute or two after today at 16:00.
>>> >>> >> >>>>
>>> >>> >> >>>>
>>> >>> >> >>>>
>>> >>> >> >>>>
>>> >>> >> >>>> https://drive.google.com/folderview?id=0Bzz8TrxFvfema2NQUmotd1BOTnM&usp=sharing
>>> >>> >> >>>>
>>> >>> >> >>>>
>>> >>> >> >>>> Oddly(to me anyways), this pg is now active+clean:
>>> >>> >> >>>>
>>> >>> >> >>>> # ceph pg dump | grep 70.459
>>> >>> >> >>>> dumped all in format plain
>>> >>> >> >>>> 70.459 21377 0 0 0 0 64515446306 3088 3088 active+clean
>>> >>> >> >>>> 2016-03-07
>>> >>> >> >>>> 16:26:57.796537 279563'212832 279602:628151
>>> >>> >> >>>> [307,210,273,191,132,450]
>>> >>> >> >>>> 307
>>> >>> >> >>>> [307,210,273,191,132,450] 307 279563'212832 2016-03-07
>>> >>> >> >>>> 16:12:30.741984
>>> >>> >> >>>> 279563'212832 2016-03-07 16:12:30.741984
>>> >>> >> >>>>
>>> >>> >> >>>>
>>> >>> >> >>>>
>>> >>> >> >>>> Regards,
>>> >>> >> >>>> Jeff
>>> >>> >> >>>>
>>> >>> >> >>>>
>>> >>> >> >>>> On Mon, Mar 7, 2016 at 4:11 PM, Samuel Just <sjust@xxxxxxxxxx>
>>> >>> >> >>>> wrote:
>>> >>> >> >>>>>
>>> >>> >> >>>>> I think the unfound object on repair is fixed by
>>> >>> >> >>>>> d51806f5b330d5f112281fbb95ea6addf994324e (not in hammer yet).
>>> >>> >> >>>>> I
>>> >>> >> >>>>> opened http://tracker.ceph.com/issues/15002 for the backport
>>> >>> >> >>>>> and
>>> >>> >> >>>>> to
>>> >>> >> >>>>> make sure it's covered in ceph-qa-suite. No idea at this
>>> >>> >> >>>>> time
>>> >>> >> >>>>> why
>>> >>> >> >>>>> the
>>> >>> >> >>>>> objects are disappearing though.
>>> >>> >> >>>>> -Sam
>>> >>> >> >>>>>
>>> >>> >> >>>>> On Mon, Mar 7, 2016 at 1:57 PM, Samuel Just
>>> >>> >> >>>>> <sjust@xxxxxxxxxx>
>>> >>> >> >>>>> wrote:
>>> >>> >> >>>>> > The one just scrubbed and now inconsistent.
>>> >>> >> >>>>> > -Sam
>>> >>> >> >>>>> >
>>> >>> >> >>>>> > On Mon, Mar 7, 2016 at 1:57 PM, Jeffrey McDonald
>>> >>> >> >>>>> > <jmcdonal@xxxxxxx>
>>> >>> >> >>>>> > wrote:
>>> >>> >> >>>>> >> Do you want me to enable this for the pg already with
>>> >>> >> >>>>> >> unfound
>>> >>> >> >>>>> >> objects
>>> >>> >> >>>>> >> or the
>>> >>> >> >>>>> >> placement group just scrubbed and now inconsistent?
>>> >>> >> >>>>> >> Jeff
>>> >>> >> >>>>> >>
>>> >>> >> >>>>> >> On Mon, Mar 7, 2016 at 3:54 PM, Samuel Just
>>> >>> >> >>>>> >> <sjust@xxxxxxxxxx>
>>> >>> >> >>>>> >> wrote:
>>> >>> >> >>>>> >>>
>>> >>> >> >>>>> >>> Can you enable
>>> >>> >> >>>>> >>>
>>> >>> >> >>>>> >>> debug osd = 20
>>> >>> >> >>>>> >>> debug filestore = 20
>>> >>> >> >>>>> >>> debug ms = 1
>>> >>> >> >>>>> >>>
>>> >>> >> >>>>> >>> on all osds in that PG, rescrub, and convey to us the
>>> >>> >> >>>>> >>> resulting
>>> >>> >> >>>>> >>> logs?
>>> >>> >> >>>>> >>> -Sam
>>> >>> >> >>>>> >>>
>>> >>> >> >>>>> >>> On Mon, Mar 7, 2016 at 1:36 PM, Jeffrey McDonald
>>> >>> >> >>>>> >>> <jmcdonal@xxxxxxx>
>>> >>> >> >>>>> >>> wrote:
>>> >>> >> >>>>> >>> > Here is a PG which just went inconsistent:
>>> >>> >> >>>>> >>> >
>>> >>> >> >>>>> >>> > pg 70.459 is active+clean+inconsistent, acting
>>> >>> >> >>>>> >>> > [307,210,273,191,132,450]
>>> >>> >> >>>>> >>> >
>>> >>> >> >>>>> >>> > Attached is the result of a pg query on this. I will
>>> >>> >> >>>>> >>> > wait
>>> >>> >> >>>>> >>> > for your
>>> >>> >> >>>>> >>> > feedback before issuing a repair.
>>> >>> >> >>>>> >>> >
>>> >>> >> >>>>> >>> > From what I read, the inconsistencies are more likely
>>> >>> >> >>>>> >>> > the
>>> >>> >> >>>>> >>> > result of
>>> >>> >> >>>>> >>> > ntp,
>>> >>> >> >>>>> >>> > but
>>> >>> >> >>>>> >>> > all nodes have the local ntp master and all are showing
>>> >>> >> >>>>> >>> > sync.
>>> >>> >> >>>>> >>> >
>>> >>> >> >>>>> >>> > Regards,
>>> >>> >> >>>>> >>> > Jeff
>>> >>> >> >>>>> >>> >
>>> >>> >> >>>>> >>> > On Mon, Mar 7, 2016 at 3:15 PM, Gregory Farnum
>>> >>> >> >>>>> >>> > <gfarnum@xxxxxxxxxx>
>>> >>> >> >>>>> >>> > wrote:
>>> >>> >> >>>>> >>> >>
>>> >>> >> >>>>> >>> >> [ Keeping this on the users list. ]
>>> >>> >> >>>>> >>> >>
>>> >>> >> >>>>> >>> >> Okay, so next time this happens you probably want to
>>> >>> >> >>>>> >>> >> do a
>>> >>> >> >>>>> >>> >> pg
>>> >>> >> >>>>> >>> >> query
>>> >>> >> >>>>> >>> >> on
>>> >>> >> >>>>> >>> >> the PG which has been reported as dirty. I can't help
>>> >>> >> >>>>> >>> >> much
>>> >>> >> >>>>> >>> >> beyond
>>> >>> >> >>>>> >>> >> that, but hopefully Kefu or David will chime in once
>>> >>> >> >>>>> >>> >> there's
>>> >>> >> >>>>> >>> >> a
>>> >>> >> >>>>> >>> >> little
>>> >>> >> >>>>> >>> >> more for them to look at.
>>> >>> >> >>>>> >>> >> -Greg
>>> >>> >> >>>>> >>> >>
>>> >>> >> >>>>> >>> >> On Mon, Mar 7, 2016 at 1:00 PM, Jeffrey McDonald
>>> >>> >> >>>>> >>> >> <jmcdonal@xxxxxxx>
>>> >>> >> >>>>> >>> >> wrote:
>>> >>> >> >>>>> >>> >> > Hi Greg,
>>> >>> >> >>>>> >>> >> >
>>> >>> >> >>>>> >>> >> > I'm running the ceph version hammer,
>>> >>> >> >>>>> >>> >> > ceph version 0.94.5
>>> >>> >> >>>>> >>> >> > (9764da52395923e0b32908d83a9f7304401fee43)
>>> >>> >> >>>>> >>> >> >
>>> >>> >> >>>>> >>> >> > The hardware migration was performed by just setting
>>> >>> >> >>>>> >>> >> > the
>>> >>> >> >>>>> >>> >> > crush
>>> >>> >> >>>>> >>> >> > map to
>>> >>> >> >>>>> >>> >> > zero
>>> >>> >> >>>>> >>> >> > for the OSD we wanted to retire. The system was
>>> >>> >> >>>>> >>> >> > performing
>>> >>> >> >>>>> >>> >> > poorly
>>> >>> >> >>>>> >>> >> > with
>>> >>> >> >>>>> >>> >> > these older OSDs and we had a difficult time
>>> >>> >> >>>>> >>> >> > maintaining
>>> >>> >> >>>>> >>> >> > stability of
>>> >>> >> >>>>> >>> >> > the
>>> >>> >> >>>>> >>> >> > system. The old OSDs are still there but all of
>>> >>> >> >>>>> >>> >> > the
>>> >>> >> >>>>> >>> >> > data
>>> >>> >> >>>>> >>> >> > is
>>> >>> >> >>>>> >>> >> > now
>>> >>> >> >>>>> >>> >> > migrated
>>> >>> >> >>>>> >>> >> > to new and/or existing hardware.
>>> >>> >> >>>>> >>> >> >
>>> >>> >> >>>>> >>> >> > Thanks,
>>> >>> >> >>>>> >>> >> > Jeff
>>> >>> >> >>>>> >>> >> >
>>> >>> >> >>>>> >>> >> >
>>> >>> >> >>>>> >>> >> >
>>> >>> >> >>>>> >>> >> >
>>> >>> >> >>>>> >>> >> >
>>> >>> >> >>>>> >>> >> > On Mon, Mar 7, 2016 at 2:56 PM, Gregory Farnum
>>> >>> >> >>>>> >>> >> > <gfarnum@xxxxxxxxxx>
>>> >>> >> >>>>> >>> >> > wrote:
>>> >>> >> >>>>> >>> >> >>
>>> >>> >> >>>>> >>> >> >> On Mon, Mar 7, 2016 at 12:07 PM, Jeffrey McDonald
>>> >>> >> >>>>> >>> >> >> <jmcdonal@xxxxxxx>
>>> >>> >> >>>>> >>> >> >> wrote:
>>> >>> >> >>>>> >>> >> >> > Hi,
>>> >>> >> >>>>> >>> >> >> >
>>> >>> >> >>>>> >>> >> >> > For a while, we've been seeing inconsistent
>>> >>> >> >>>>> >>> >> >> > placement
>>> >>> >> >>>>> >>> >> >> > groups
>>> >>> >> >>>>> >>> >> >> > on
>>> >>> >> >>>>> >>> >> >> > our
>>> >>> >> >>>>> >>> >> >> > erasure
>>> >>> >> >>>>> >>> >> >> > coded system. The placement groups go from a
>>> >>> >> >>>>> >>> >> >> > state
>>> >>> >> >>>>> >>> >> >> > of
>>> >>> >> >>>>> >>> >> >> > active+clean
>>> >>> >> >>>>> >>> >> >> > to
>>> >>> >> >>>>> >>> >> >> > active+clean+inconsistent after a deep scrub:
>>> >>> >> >>>>> >>> >> >> >
>>> >>> >> >>>>> >>> >> >> >
>>> >>> >> >>>>> >>> >> >> > 2016-03-07 13:45:42.044131 7f385d118700 -1
>>> >>> >> >>>>> >>> >> >> > log_channel(cluster)
>>> >>> >> >>>>> >>> >> >> > log
>>> >>> >> >>>>> >>> >> >> > [ERR] :
>>> >>> >> >>>>> >>> >> >> > 70.320s0 deep-scrub stat mismatch, got
>>> >>> >> >>>>> >>> >> >> > 21446/21428
>>> >>> >> >>>>> >>> >> >> > objects,
>>> >>> >> >>>>> >>> >> >> > 0/0
>>> >>> >> >>>>> >>> >> >> > clones,
>>> >>> >> >>>>> >>> >> >> > 21446/21428 dirty, 0/0 omap, 0/0 hit_set_archive,
>>> >>> >> >>>>> >>> >> >> > 0/0
>>> >>> >> >>>>> >>> >> >> > whiteouts,
>>> >>> >> >>>>> >>> >> >> > 64682334170/64624353083 bytes,0/0 hit_set_archive
>>> >>> >> >>>>> >>> >> >> > bytes.
>>> >>> >> >>>>> >>> >> >> > 2016-03-07 13:45:42.044416 7f385d118700 -1
>>> >>> >> >>>>> >>> >> >> > log_channel(cluster)
>>> >>> >> >>>>> >>> >> >> > log
>>> >>> >> >>>>> >>> >> >> > [ERR] :
>>> >>> >> >>>>> >>> >> >> > 70.320s0 deep-scrub 18 missing, 0 inconsistent
>>> >>> >> >>>>> >>> >> >> > objects
>>> >>> >> >>>>> >>> >> >> > 2016-03-07 13:45:42.044464 7f385d118700 -1
>>> >>> >> >>>>> >>> >> >> > log_channel(cluster)
>>> >>> >> >>>>> >>> >> >> > log
>>> >>> >> >>>>> >>> >> >> > [ERR] :
>>> >>> >> >>>>> >>> >> >> > 70.320 deep-scrub 73 errors
>>> >>> >> >>>>> >>> >> >> >
>>> >>> >> >>>>> >>> >> >> > So I tell the placement group to perform a
>>> >>> >> >>>>> >>> >> >> > repair:
>>> >>> >> >>>>> >>> >> >> >
>>> >>> >> >>>>> >>> >> >> > 2016-03-07 13:49:26.047177 7f385d118700 0
>>> >>> >> >>>>> >>> >> >> > log_channel(cluster)
>>> >>> >> >>>>> >>> >> >> > log
>>> >>> >> >>>>> >>> >> >> > [INF] :
>>> >>> >> >>>>> >>> >> >> > 70.320 repair starts
>>> >>> >> >>>>> >>> >> >> > 2016-03-07 13:49:57.087291 7f3858b0a700 0 --
>>> >>> >> >>>>> >>> >> >> > 10.31.0.2:6874/13937
>>> >>> >> >>>>> >>> >> >> > >>
>>> >>> >> >>>>> >>> >> >> > 10.31.0.6:6824/8127 pipe(0x2e578000 sd=697 :6874
>>> >>> >> >>>>> >>> >> >> >
>>> >>> >> >>>>> >>> >> >> > The repair finds missing shards and repairs them,
>>> >>> >> >>>>> >>> >> >> > but
>>> >>> >> >>>>> >>> >> >> > then I
>>> >>> >> >>>>> >>> >> >> > have
>>> >>> >> >>>>> >>> >> >> > 18
>>> >>> >> >>>>> >>> >> >> > 'unfound objects' :
>>> >>> >> >>>>> >>> >> >> >
>>> >>> >> >>>>> >>> >> >> >
>>> >>> >> >>>>> >>> >> >> > 2016-03-07 13:51:28.467590 7f385d118700 -1
>>> >>> >> >>>>> >>> >> >> > log_channel(cluster)
>>> >>> >> >>>>> >>> >> >> > log
>>> >>> >> >>>>> >>> >> >> > [ERR] :
>>> >>> >> >>>>> >>> >> >> > 70.320s0 repair stat mismatch, got 21446/21428
>>> >>> >> >>>>> >>> >> >> > objects,
>>> >>> >> >>>>> >>> >> >> > 0/0
>>> >>> >> >>>>> >>> >> >> > clones,
>>> >>> >> >>>>> >>> >> >> > 21446/21428 dirty, 0/0 omap, 0/0 hit_set_archive,
>>> >>> >> >>>>> >>> >> >> > 0/0
>>> >>> >> >>>>> >>> >> >> > whiteouts,
>>> >>> >> >>>>> >>> >> >> > 64682334170/64624353083 bytes,0/0 hit_set_archive
>>> >>> >> >>>>> >>> >> >> > bytes.
>>> >>> >> >>>>> >>> >> >> > 2016-03-07 13:51:28.468358 7f385d118700 -1
>>> >>> >> >>>>> >>> >> >> > log_channel(cluster)
>>> >>> >> >>>>> >>> >> >> > log
>>> >>> >> >>>>> >>> >> >> > [ERR] :
>>> >>> >> >>>>> >>> >> >> > 70.320s0 repair 18 missing, 0 inconsistent
>>> >>> >> >>>>> >>> >> >> > objects
>>> >>> >> >>>>> >>> >> >> > 2016-03-07 13:51:28.469431 7f385d118700 -1
>>> >>> >> >>>>> >>> >> >> > log_channel(cluster)
>>> >>> >> >>>>> >>> >> >> > log
>>> >>> >> >>>>> >>> >> >> > [ERR] :
>>> >>> >> >>>>> >>> >> >> > 70.320 repair 73 errors, 73 fixed
>>> >>> >> >>>>> >>> >> >> >
>>> >>> >> >>>>> >>> >> >> >
>>> >>> >> >>>>> >>> >> >> > I've traced one of the unfound objects all the
>>> >>> >> >>>>> >>> >> >> > way
>>> >>> >> >>>>> >>> >> >> > through the
>>> >>> >> >>>>> >>> >> >> > system
>>> >>> >> >>>>> >>> >> >> > and
>>> >>> >> >>>>> >>> >> >> > I've found that they are not really lost. I can
>>> >>> >> >>>>> >>> >> >> > fail
>>> >>> >> >>>>> >>> >> >> > over
>>> >>> >> >>>>> >>> >> >> > the
>>> >>> >> >>>>> >>> >> >> > osd
>>> >>> >> >>>>> >>> >> >> > and
>>> >>> >> >>>>> >>> >> >> > recover the files. This is happening quite
>>> >>> >> >>>>> >>> >> >> > regularly
>>> >>> >> >>>>> >>> >> >> > now
>>> >>> >> >>>>> >>> >> >> > after a
>>> >>> >> >>>>> >>> >> >> > large
>>> >>> >> >>>>> >>> >> >> > migration of data from old hardware to
>>> >>> >> >>>>> >>> >> >> > new(migration
>>> >>> >> >>>>> >>> >> >> > is
>>> >>> >> >>>>> >>> >> >> > now
>>> >>> >> >>>>> >>> >> >> > complete).
>>> >>> >> >>>>> >>> >> >> >
>>> >>> >> >>>>> >>> >> >> > The system sets the PG into 'recovery', but we've
>>> >>> >> >>>>> >>> >> >> > seen
>>> >>> >> >>>>> >>> >> >> > the
>>> >>> >> >>>>> >>> >> >> > system
>>> >>> >> >>>>> >>> >> >> > in
>>> >>> >> >>>>> >>> >> >> > a
>>> >>> >> >>>>> >>> >> >> > recovering state for many days. Should we just
>>> >>> >> >>>>> >>> >> >> > be
>>> >>> >> >>>>> >>> >> >> > patient
>>> >>> >> >>>>> >>> >> >> > or do
>>> >>> >> >>>>> >>> >> >> > we
>>> >>> >> >>>>> >>> >> >> > need
>>> >>> >> >>>>> >>> >> >> > to dig further into the issue?
>>> >>> >> >>>>> >>> >> >>
>>> >>> >> >>>>> >>> >> >> You may need to dig into this more, although I'm
>>> >>> >> >>>>> >>> >> >> not
>>> >>> >> >>>>> >>> >> >> sure
>>> >>> >> >>>>> >>> >> >> what
>>> >>> >> >>>>> >>> >> >> the
>>> >>> >> >>>>> >>> >> >> issue is likely to be. What version of Ceph are you
>>> >>> >> >>>>> >>> >> >> running? How
>>> >>> >> >>>>> >>> >> >> did
>>> >>> >> >>>>> >>> >> >> you do this hardware migration?
>>> >>> >> >>>>> >>> >> >> -Greg
>>> >>> >> >>>>> >>> >> >>
>>> >>> >> >>>>> >>> >> >> >
>>> >>> >> >>>>> >>> >> >> >
>>> >>> >> >>>>> >>> >> >> > pg 70.320 is stuck unclean for 704.803040,
>>> >>> >> >>>>> >>> >> >> > current
>>> >>> >> >>>>> >>> >> >> > state
>>> >>> >> >>>>> >>> >> >> > active+recovering,
>>> >>> >> >>>>> >>> >> >> > last acting [277,101,218,49,304,412]
>>> >>> >> >>>>> >>> >> >> > pg 70.320 is active+recovering, acting
>>> >>> >> >>>>> >>> >> >> > [277,101,218,49,304,412],
>>> >>> >> >>>>> >>> >> >> > 18
>>> >>> >> >>>>> >>> >> >> > unfound
>>> >>> >> >>>>> >>> >> >> >
>>> >>> >> >>>>> >>> >> >> > There is no indication of any problems with down
>>> >>> >> >>>>> >>> >> >> > OSDs
>>> >>> >> >>>>> >>> >> >> > or
>>> >>> >> >>>>> >>> >> >> > network
>>> >>> >> >>>>> >>> >> >> > issues
>>> >>> >> >>>>> >>> >> >> > with
>>> >>> >> >>>>> >>> >> >> > OSDs.
>>> >>> >> >>>>> >>> >> >> >
>>> >>> >> >>>>> >>> >> >> > Thanks,
>>> >>> >> >>>>> >>> >> >> > Jeff
>>> >>> >> >>>>> >>> >> >> >
>>> >>> >> >>>>> >>> >> >> >
>>> >>> >> >>>>> >>> >> >> > --
>>> >>> >> >>>>> >>> >> >> >
>>> >>> >> >>>>> >>> >> >> > Jeffrey McDonald, PhD
>>> >>> >> >>>>> >>> >> >> > Assistant Director for HPC Operations
>>> >>> >> >>>>> >>> >> >> > Minnesota Supercomputing Institute
>>> >>> >> >>>>> >>> >> >> > University of Minnesota Twin Cities
>>> >>> >> >>>>> >>> >> >> > 599 Walter Library email:
>>> >>> >> >>>>> >>> >> >> > jeffrey.mcdonald@xxxxxxxxxxx
>>> >>> >> >>>>> >>> >> >> > 117 Pleasant St SE phone: +1 612
>>> >>> >> >>>>> >>> >> >> > 625-6905
>>> >>> >> >>>>> >>> >> >> > Minneapolis, MN 55455 fax: +1 612
>>> >>> >> >>>>> >>> >> >> > 624-8861
>>> >>> >> >>>>> >>> >> >> >
>>> >>> >> >>>>> >>> >> >> >
>>> >>> >> >>>>> >>> >> >> >
>>> >>> >> >>>>> >>> >> >> > _______________________________________________
>>> >>> >> >>>>> >>> >> >> > ceph-users mailing list
>>> >>> >> >>>>> >>> >> >> > ceph-users@xxxxxxxxxxxxxx
>>> >>> >> >>>>> >>> >> >> >
>>> >>> >> >>>>> >>> >> >> >
>>> >>> >> >>>>> >>> >> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> >>> >> >>>>> >>> >> >> >
>>> >>> >> >>>>> >>> >> >
>>> >>> >> >>>>> >>> >> >
>>> >>> >> >>>>> >>> >> >
>>> >>> >> >>>>> >>> >> >
>>> >>> >> >>>>> >>> >> > --
>>> >>> >> >>>>> >>> >> >
>>> >>> >> >>>>> >>> >> > Jeffrey McDonald, PhD
>>> >>> >> >>>>> >>> >> > Assistant Director for HPC Operations
>>> >>> >> >>>>> >>> >> > Minnesota Supercomputing Institute
>>> >>> >> >>>>> >>> >> > University of Minnesota Twin Cities
>>> >>> >> >>>>> >>> >> > 599 Walter Library email:
>>> >>> >> >>>>> >>> >> > jeffrey.mcdonald@xxxxxxxxxxx
>>> >>> >> >>>>> >>> >> > 117 Pleasant St SE phone: +1 612 625-6905
>>> >>> >> >>>>> >>> >> > Minneapolis, MN 55455 fax: +1 612 624-8861
>>> >>> >> >>>>> >>> >> >
>>> >>> >> >>>>> >>> >> >
>>> >>> >> >>>>> >>> >
>>> >>> >> >>>>> >>> >
>>> >>> >> >>>>> >>> >
>>> >>> >> >>>>> >>> >
>>> >>> >> >>>>> >>> > --
>>> >>> >> >>>>> >>> >
>>> >>> >> >>>>> >>> > Jeffrey McDonald, PhD
>>> >>> >> >>>>> >>> > Assistant Director for HPC Operations
>>> >>> >> >>>>> >>> > Minnesota Supercomputing Institute
>>> >>> >> >>>>> >>> > University of Minnesota Twin Cities
>>> >>> >> >>>>> >>> > 599 Walter Library email:
>>> >>> >> >>>>> >>> > jeffrey.mcdonald@xxxxxxxxxxx
>>> >>> >> >>>>> >>> > 117 Pleasant St SE phone: +1 612 625-6905
>>> >>> >> >>>>> >>> > Minneapolis, MN 55455 fax: +1 612 624-8861
>>> >>> >> >>>>> >>> >
>>> >>> >> >>>>> >>> >
>>> >>> >> >>>>> >>> >
>>> >>> >> >>>>> >>> > _______________________________________________
>>> >>> >> >>>>> >>> > ceph-users mailing list
>>> >>> >> >>>>> >>> > ceph-users@xxxxxxxxxxxxxx
>>> >>> >> >>>>> >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> >>> >> >>>>> >>> >
>>> >>> >> >>>>> >>
>>> >>> >> >>>>> >>
>>> >>> >> >>>>> >>
>>> >>> >> >>>>> >>
>>> >>> >> >>>>> >> --
>>> >>> >> >>>>> >>
>>> >>> >> >>>>> >> Jeffrey McDonald, PhD
>>> >>> >> >>>>> >> Assistant Director for HPC Operations
>>> >>> >> >>>>> >> Minnesota Supercomputing Institute
>>> >>> >> >>>>> >> University of Minnesota Twin Cities
>>> >>> >> >>>>> >> 599 Walter Library email:
>>> >>> >> >>>>> >> jeffrey.mcdonald@xxxxxxxxxxx
>>> >>> >> >>>>> >> 117 Pleasant St SE phone: +1 612 625-6905
>>> >>> >> >>>>> >> Minneapolis, MN 55455 fax: +1 612 624-8861
>>> >>> >> >>>>> >>
>>> >>> >> >>>>> >>
>>> >>> >> >>>>
>>> >>> >> >>>>
>>> >>> >> >>>>
>>> >>> >> >>>>
>>> >>> >> >>>> --
>>> >>> >> >>>>
>>> >>> >> >>>> Jeffrey McDonald, PhD
>>> >>> >> >>>> Assistant Director for HPC Operations
>>> >>> >> >>>> Minnesota Supercomputing Institute
>>> >>> >> >>>> University of Minnesota Twin Cities
>>> >>> >> >>>> 599 Walter Library email:
>>> >>> >> >>>> jeffrey.mcdonald@xxxxxxxxxxx
>>> >>> >> >>>> 117 Pleasant St SE phone: +1 612 625-6905
>>> >>> >> >>>> Minneapolis, MN 55455 fax: +1 612 624-8861
>>> >>> >> >>>>
>>> >>> >> >>>>
>>> >>> >> > _______________________________________________
>>> >>> >> > ceph-users mailing list
>>> >>> >> > ceph-users@xxxxxxxxxxxxxx
>>> >>> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> >>> >>
>>> >>> >>
>>> >>> >>
>>> >>> >> --
>>> >>> >> Email:
>>> >>> >> shinobu@xxxxxxxxx
>>> >>> >> GitHub:
>>> >>> >> shinobu-x
>>> >>> >> Blog:
>>> >>> >> Life with Distributed Computational System based on OpenSource
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> > --
>>> >>> >
>>> >>> > Jeffrey McDonald, PhD
>>> >>> > Assistant Director for HPC Operations
>>> >>> > Minnesota Supercomputing Institute
>>> >>> > University of Minnesota Twin Cities
>>> >>> > 599 Walter Library email: jeffrey.mcdonald@xxxxxxxxxxx
>>> >>> > 117 Pleasant St SE phone: +1 612 625-6905
>>> >>> > Minneapolis, MN 55455 fax: +1 612 624-8861
>>> >>> >
>>> >>> >
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >>
>>> >> Jeffrey McDonald, PhD
>>> >> Assistant Director for HPC Operations
>>> >> Minnesota Supercomputing Institute
>>> >> University of Minnesota Twin Cities
>>> >> 599 Walter Library email: jeffrey.mcdonald@xxxxxxxxxxx
>>> >> 117 Pleasant St SE phone: +1 612 625-6905
>>> >> Minneapolis, MN 55455 fax: +1 612 624-8861
>>> >>
>>> >>
>>
>>
>>
>>
>> --
>>
>> Jeffrey McDonald, PhD
>> Assistant Director for HPC Operations
>> Minnesota Supercomputing Institute
>> University of Minnesota Twin Cities
>> 599 Walter Library email: jeffrey.mcdonald@xxxxxxxxxxx
>> 117 Pleasant St SE phone: +1 612 625-6905
>> Minneapolis, MN 55455 fax: +1 612 624-8861
>>
>>
Jeffrey McDonald, PhD Assistant Director for HPC Operations Minnesota Supercomputing Institute University of Minnesota Twin Cities 599 Walter Library email: jeffrey.mcdonald@xxxxxxxxxxx 117 Pleasant St SE phone: +1 612 625-6905 Minneapolis, MN 55455 fax: +1 612 624-8861
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com