I have one other very strange event happening, I've opened a ticket on it:
During this migration, OSD failed probably over 400 times while moving data around. We move the empty directories and restarted the OSDs. I can't say if this is related--I have no reason to suspect it is.
Jeff
On Mon, Mar 7, 2016 at 5:31 PM, Shinobu Kinjo <shinobu.kj@xxxxxxxxx> wrote:
What could cause this kind of unexpected behaviour?
Any assumption??
Sorry for interrupting you.
Cheers,
S
Email:
On Tue, Mar 8, 2016 at 8:19 AM, Samuel Just <sjust@xxxxxxxxxx> wrote:
> Hmm, at the end of the log, the pg is still inconsistent. Can you
> attach a ceph pg query on that pg?
> -Sam
>
> On Mon, Mar 7, 2016 at 3:05 PM, Samuel Just <sjust@xxxxxxxxxx> wrote:
>> If so, that strongly suggests that the pg was actually never
>> inconsistent in the first place and that the bug is in scrub itself
>> presumably getting confused about an object during a write. The next
>> step would be to get logs like the above from a pg as it scrubs
>> transitioning from clean to inconsistent. If it's really a race
>> between scrub and a write, it's probably just non-deterministic, you
>> could set logging on a set of osds and continuously scrub any pgs
>> which only map to those osds until you reproduce the problem.
>> -Sam
>>
>> On Mon, Mar 7, 2016 at 2:44 PM, Samuel Just <sjust@xxxxxxxxxx> wrote:
>>> So after the scrub, it came up clean? The inconsistent/missing
>>> objects reappeared?
>>> -Sam
>>>
>>> On Mon, Mar 7, 2016 at 2:33 PM, Jeffrey McDonald <jmcdonal@xxxxxxx> wrote:
>>>> Hi Sam,
>>>>
>>>> I've done as you requested:
>>>>
>>>> pg 70.459 is active+clean+inconsistent, acting [307,210,273,191,132,450]
>>>>
>>>> # for i in 307 210 273 191 132 450 ; do
>>>>> ceph tell osd.$i injectargs '--debug-osd 20 --debug-filestore 20
>>>>> --debug-ms 1'
>>>>> done
>>>> debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1
>>>> debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1
>>>> debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1
>>>> debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1
>>>> debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1
>>>> debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1
>>>>
>>>>
>>>> # date
>>>> Mon Mar 7 16:03:38 CST 2016
>>>>
>>>>
>>>> # ceph pg deep-scrub 70.459
>>>> instructing pg 70.459 on osd.307 to deep-scrub
>>>>
>>>>
>>>>
>>>> Scrub finished around
>>>>
>>>> # date
>>>> Mon Mar 7 16:13:03 CST 2016
>>>>
>>>>
>>>>
>>>>
>>>> I've tar'd+gziped the files which can be downloaded from here. The logs
>>>> start a minute or two after today at 16:00.
>>>>
>>>> https://drive.google.com/folderview?id=0Bzz8TrxFvfema2NQUmotd1BOTnM&usp=sharing
>>>>
>>>>
>>>> Oddly(to me anyways), this pg is now active+clean:
>>>>
>>>> # ceph pg dump | grep 70.459
>>>> dumped all in format plain
>>>> 70.459 21377 0 0 0 0 64515446306 3088 3088 active+clean 2016-03-07
>>>> 16:26:57.796537 279563'212832 279602:628151 [307,210,273,191,132,450] 307
>>>> [307,210,273,191,132,450] 307 279563'212832 2016-03-07 16:12:30.741984
>>>> 279563'212832 2016-03-07 16:12:30.741984
>>>>
>>>>
>>>>
>>>> Regards,
>>>> Jeff
>>>>
>>>>
>>>> On Mon, Mar 7, 2016 at 4:11 PM, Samuel Just <sjust@xxxxxxxxxx> wrote:
>>>>>
>>>>> I think the unfound object on repair is fixed by
>>>>> d51806f5b330d5f112281fbb95ea6addf994324e (not in hammer yet). I
>>>>> opened http://tracker.ceph.com/issues/15002 for the backport and to
>>>>> make sure it's covered in ceph-qa-suite. No idea at this time why the
>>>>> objects are disappearing though.
>>>>> -Sam
>>>>>
>>>>> On Mon, Mar 7, 2016 at 1:57 PM, Samuel Just <sjust@xxxxxxxxxx> wrote:
>>>>> > The one just scrubbed and now inconsistent.
>>>>> > -Sam
>>>>> >
>>>>> > On Mon, Mar 7, 2016 at 1:57 PM, Jeffrey McDonald <jmcdonal@xxxxxxx>
>>>>> > wrote:
>>>>> >> Do you want me to enable this for the pg already with unfound objects
>>>>> >> or the
>>>>> >> placement group just scrubbed and now inconsistent?
>>>>> >> Jeff
>>>>> >>
>>>>> >> On Mon, Mar 7, 2016 at 3:54 PM, Samuel Just <sjust@xxxxxxxxxx> wrote:
>>>>> >>>
>>>>> >>> Can you enable
>>>>> >>>
>>>>> >>> debug osd = 20
>>>>> >>> debug filestore = 20
>>>>> >>> debug ms = 1
>>>>> >>>
>>>>> >>> on all osds in that PG, rescrub, and convey to us the resulting logs?
>>>>> >>> -Sam
>>>>> >>>
>>>>> >>> On Mon, Mar 7, 2016 at 1:36 PM, Jeffrey McDonald <jmcdonal@xxxxxxx>
>>>>> >>> wrote:
>>>>> >>> > Here is a PG which just went inconsistent:
>>>>> >>> >
>>>>> >>> > pg 70.459 is active+clean+inconsistent, acting
>>>>> >>> > [307,210,273,191,132,450]
>>>>> >>> >
>>>>> >>> > Attached is the result of a pg query on this. I will wait for your
>>>>> >>> > feedback before issuing a repair.
>>>>> >>> >
>>>>> >>> > From what I read, the inconsistencies are more likely the result of
>>>>> >>> > ntp,
>>>>> >>> > but
>>>>> >>> > all nodes have the local ntp master and all are showing sync.
>>>>> >>> >
>>>>> >>> > Regards,
>>>>> >>> > Jeff
>>>>> >>> >
>>>>> >>> > On Mon, Mar 7, 2016 at 3:15 PM, Gregory Farnum <gfarnum@xxxxxxxxxx>
>>>>> >>> > wrote:
>>>>> >>> >>
>>>>> >>> >> [ Keeping this on the users list. ]
>>>>> >>> >>
>>>>> >>> >> Okay, so next time this happens you probably want to do a pg query
>>>>> >>> >> on
>>>>> >>> >> the PG which has been reported as dirty. I can't help much beyond
>>>>> >>> >> that, but hopefully Kefu or David will chime in once there's a
>>>>> >>> >> little
>>>>> >>> >> more for them to look at.
>>>>> >>> >> -Greg
>>>>> >>> >>
>>>>> >>> >> On Mon, Mar 7, 2016 at 1:00 PM, Jeffrey McDonald <jmcdonal@xxxxxxx>
>>>>> >>> >> wrote:
>>>>> >>> >> > Hi Greg,
>>>>> >>> >> >
>>>>> >>> >> > I'm running the ceph version hammer,
>>>>> >>> >> > ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)
>>>>> >>> >> >
>>>>> >>> >> > The hardware migration was performed by just setting the crush
>>>>> >>> >> > map to
>>>>> >>> >> > zero
>>>>> >>> >> > for the OSD we wanted to retire. The system was performing
>>>>> >>> >> > poorly
>>>>> >>> >> > with
>>>>> >>> >> > these older OSDs and we had a difficult time maintaining
>>>>> >>> >> > stability of
>>>>> >>> >> > the
>>>>> >>> >> > system. The old OSDs are still there but all of the data is
>>>>> >>> >> > now
>>>>> >>> >> > migrated
>>>>> >>> >> > to new and/or existing hardware.
>>>>> >>> >> >
>>>>> >>> >> > Thanks,
>>>>> >>> >> > Jeff
>>>>> >>> >> >
>>>>> >>> >> >
>>>>> >>> >> >
>>>>> >>> >> >
>>>>> >>> >> >
>>>>> >>> >> > On Mon, Mar 7, 2016 at 2:56 PM, Gregory Farnum
>>>>> >>> >> > <gfarnum@xxxxxxxxxx>
>>>>> >>> >> > wrote:
>>>>> >>> >> >>
>>>>> >>> >> >> On Mon, Mar 7, 2016 at 12:07 PM, Jeffrey McDonald
>>>>> >>> >> >> <jmcdonal@xxxxxxx>
>>>>> >>> >> >> wrote:
>>>>> >>> >> >> > Hi,
>>>>> >>> >> >> >
>>>>> >>> >> >> > For a while, we've been seeing inconsistent placement groups
>>>>> >>> >> >> > on
>>>>> >>> >> >> > our
>>>>> >>> >> >> > erasure
>>>>> >>> >> >> > coded system. The placement groups go from a state of
>>>>> >>> >> >> > active+clean
>>>>> >>> >> >> > to
>>>>> >>> >> >> > active+clean+inconsistent after a deep scrub:
>>>>> >>> >> >> >
>>>>> >>> >> >> >
>>>>> >>> >> >> > 2016-03-07 13:45:42.044131 7f385d118700 -1
>>>>> >>> >> >> > log_channel(cluster)
>>>>> >>> >> >> > log
>>>>> >>> >> >> > [ERR] :
>>>>> >>> >> >> > 70.320s0 deep-scrub stat mismatch, got 21446/21428 objects,
>>>>> >>> >> >> > 0/0
>>>>> >>> >> >> > clones,
>>>>> >>> >> >> > 21446/21428 dirty, 0/0 omap, 0/0 hit_set_archive, 0/0
>>>>> >>> >> >> > whiteouts,
>>>>> >>> >> >> > 64682334170/64624353083 bytes,0/0 hit_set_archive bytes.
>>>>> >>> >> >> > 2016-03-07 13:45:42.044416 7f385d118700 -1
>>>>> >>> >> >> > log_channel(cluster)
>>>>> >>> >> >> > log
>>>>> >>> >> >> > [ERR] :
>>>>> >>> >> >> > 70.320s0 deep-scrub 18 missing, 0 inconsistent objects
>>>>> >>> >> >> > 2016-03-07 13:45:42.044464 7f385d118700 -1
>>>>> >>> >> >> > log_channel(cluster)
>>>>> >>> >> >> > log
>>>>> >>> >> >> > [ERR] :
>>>>> >>> >> >> > 70.320 deep-scrub 73 errors
>>>>> >>> >> >> >
>>>>> >>> >> >> > So I tell the placement group to perform a repair:
>>>>> >>> >> >> >
>>>>> >>> >> >> > 2016-03-07 13:49:26.047177 7f385d118700 0
>>>>> >>> >> >> > log_channel(cluster)
>>>>> >>> >> >> > log
>>>>> >>> >> >> > [INF] :
>>>>> >>> >> >> > 70.320 repair starts
>>>>> >>> >> >> > 2016-03-07 13:49:57.087291 7f3858b0a700 0 --
>>>>> >>> >> >> > 10.31.0.2:6874/13937
>>>>> >>> >> >> > >>
>>>>> >>> >> >> > 10.31.0.6:6824/8127 pipe(0x2e578000 sd=697 :6874
>>>>> >>> >> >> >
>>>>> >>> >> >> > The repair finds missing shards and repairs them, but then I
>>>>> >>> >> >> > have
>>>>> >>> >> >> > 18
>>>>> >>> >> >> > 'unfound objects' :
>>>>> >>> >> >> >
>>>>> >>> >> >> >
>>>>> >>> >> >> > 2016-03-07 13:51:28.467590 7f385d118700 -1
>>>>> >>> >> >> > log_channel(cluster)
>>>>> >>> >> >> > log
>>>>> >>> >> >> > [ERR] :
>>>>> >>> >> >> > 70.320s0 repair stat mismatch, got 21446/21428 objects, 0/0
>>>>> >>> >> >> > clones,
>>>>> >>> >> >> > 21446/21428 dirty, 0/0 omap, 0/0 hit_set_archive, 0/0
>>>>> >>> >> >> > whiteouts,
>>>>> >>> >> >> > 64682334170/64624353083 bytes,0/0 hit_set_archive bytes.
>>>>> >>> >> >> > 2016-03-07 13:51:28.468358 7f385d118700 -1
>>>>> >>> >> >> > log_channel(cluster)
>>>>> >>> >> >> > log
>>>>> >>> >> >> > [ERR] :
>>>>> >>> >> >> > 70.320s0 repair 18 missing, 0 inconsistent objects
>>>>> >>> >> >> > 2016-03-07 13:51:28.469431 7f385d118700 -1
>>>>> >>> >> >> > log_channel(cluster)
>>>>> >>> >> >> > log
>>>>> >>> >> >> > [ERR] :
>>>>> >>> >> >> > 70.320 repair 73 errors, 73 fixed
>>>>> >>> >> >> >
>>>>> >>> >> >> >
>>>>> >>> >> >> > I've traced one of the unfound objects all the way through the
>>>>> >>> >> >> > system
>>>>> >>> >> >> > and
>>>>> >>> >> >> > I've found that they are not really lost. I can fail over
>>>>> >>> >> >> > the
>>>>> >>> >> >> > osd
>>>>> >>> >> >> > and
>>>>> >>> >> >> > recover the files. This is happening quite regularly now
>>>>> >>> >> >> > after a
>>>>> >>> >> >> > large
>>>>> >>> >> >> > migration of data from old hardware to new(migration is now
>>>>> >>> >> >> > complete).
>>>>> >>> >> >> >
>>>>> >>> >> >> > The system sets the PG into 'recovery', but we've seen the
>>>>> >>> >> >> > system
>>>>> >>> >> >> > in
>>>>> >>> >> >> > a
>>>>> >>> >> >> > recovering state for many days. Should we just be patient
>>>>> >>> >> >> > or do
>>>>> >>> >> >> > we
>>>>> >>> >> >> > need
>>>>> >>> >> >> > to dig further into the issue?
>>>>> >>> >> >>
>>>>> >>> >> >> You may need to dig into this more, although I'm not sure what
>>>>> >>> >> >> the
>>>>> >>> >> >> issue is likely to be. What version of Ceph are you running? How
>>>>> >>> >> >> did
>>>>> >>> >> >> you do this hardware migration?
>>>>> >>> >> >> -Greg
>>>>> >>> >> >>
>>>>> >>> >> >> >
>>>>> >>> >> >> >
>>>>> >>> >> >> > pg 70.320 is stuck unclean for 704.803040, current state
>>>>> >>> >> >> > active+recovering,
>>>>> >>> >> >> > last acting [277,101,218,49,304,412]
>>>>> >>> >> >> > pg 70.320 is active+recovering, acting
>>>>> >>> >> >> > [277,101,218,49,304,412],
>>>>> >>> >> >> > 18
>>>>> >>> >> >> > unfound
>>>>> >>> >> >> >
>>>>> >>> >> >> > There is no indication of any problems with down OSDs or
>>>>> >>> >> >> > network
>>>>> >>> >> >> > issues
>>>>> >>> >> >> > with
>>>>> >>> >> >> > OSDs.
>>>>> >>> >> >> >
>>>>> >>> >> >> > Thanks,
>>>>> >>> >> >> > Jeff
>>>>> >>> >> >> >
>>>>> >>> >> >> >
>>>>> >>> >> >> > --
>>>>> >>> >> >> >
>>>>> >>> >> >> > Jeffrey McDonald, PhD
>>>>> >>> >> >> > Assistant Director for HPC Operations
>>>>> >>> >> >> > Minnesota Supercomputing Institute
>>>>> >>> >> >> > University of Minnesota Twin Cities
>>>>> >>> >> >> > 599 Walter Library email:
>>>>> >>> >> >> > jeffrey.mcdonald@xxxxxxxxxxx
>>>>> >>> >> >> > 117 Pleasant St SE phone: +1 612 625-6905
>>>>> >>> >> >> > Minneapolis, MN 55455 fax: +1 612 624-8861
>>>>> >>> >> >> >
>>>>> >>> >> >> >
>>>>> >>> >> >> >
>>>>> >>> >> >> > _______________________________________________
>>>>> >>> >> >> > ceph-users mailing list
>>>>> >>> >> >> > ceph-users@xxxxxxxxxxxxxx
>>>>> >>> >> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>> >>> >> >> >
>>>>> >>> >> >
>>>>> >>> >> >
>>>>> >>> >> >
>>>>> >>> >> >
>>>>> >>> >> > --
>>>>> >>> >> >
>>>>> >>> >> > Jeffrey McDonald, PhD
>>>>> >>> >> > Assistant Director for HPC Operations
>>>>> >>> >> > Minnesota Supercomputing Institute
>>>>> >>> >> > University of Minnesota Twin Cities
>>>>> >>> >> > 599 Walter Library email: jeffrey.mcdonald@xxxxxxxxxxx
>>>>> >>> >> > 117 Pleasant St SE phone: +1 612 625-6905
>>>>> >>> >> > Minneapolis, MN 55455 fax: +1 612 624-8861
>>>>> >>> >> >
>>>>> >>> >> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> > --
>>>>> >>> >
>>>>> >>> > Jeffrey McDonald, PhD
>>>>> >>> > Assistant Director for HPC Operations
>>>>> >>> > Minnesota Supercomputing Institute
>>>>> >>> > University of Minnesota Twin Cities
>>>>> >>> > 599 Walter Library email: jeffrey.mcdonald@xxxxxxxxxxx
>>>>> >>> > 117 Pleasant St SE phone: +1 612 625-6905
>>>>> >>> > Minneapolis, MN 55455 fax: +1 612 624-8861
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >
>>>>> >>> > _______________________________________________
>>>>> >>> > ceph-users mailing list
>>>>> >>> > ceph-users@xxxxxxxxxxxxxx
>>>>> >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>> >>> >
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >>
>>>>> >> Jeffrey McDonald, PhD
>>>>> >> Assistant Director for HPC Operations
>>>>> >> Minnesota Supercomputing Institute
>>>>> >> University of Minnesota Twin Cities
>>>>> >> 599 Walter Library email: jeffrey.mcdonald@xxxxxxxxxxx
>>>>> >> 117 Pleasant St SE phone: +1 612 625-6905
>>>>> >> Minneapolis, MN 55455 fax: +1 612 624-8861
>>>>> >>
>>>>> >>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Jeffrey McDonald, PhD
>>>> Assistant Director for HPC Operations
>>>> Minnesota Supercomputing Institute
>>>> University of Minnesota Twin Cities
>>>> 599 Walter Library email: jeffrey.mcdonald@xxxxxxxxxxx
>>>> 117 Pleasant St SE phone: +1 612 625-6905
>>>> Minneapolis, MN 55455 fax: +1 612 624-8861
>>>>
>>>>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
shinobu@xxxxxxxxx
GitHub:
shinobu-x
Blog:
Life with Distributed Computational System based on OpenSource
Jeffrey McDonald, PhD Assistant Director for HPC Operations Minnesota Supercomputing Institute University of Minnesota Twin Cities 599 Walter Library email: jeffrey.mcdonald@xxxxxxxxxxx 117 Pleasant St SE phone: +1 612 625-6905 Minneapolis, MN 55455 fax: +1 612 624-8861
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
