Hi Sam,
I've done as you requested:
pg 70.459 is active+clean+inconsistent, acting [307,210,273,191,132,450]
# for i in 307 210 273 191 132 450 ; do
> ceph tell osd.$i injectargs '--debug-osd 20 --debug-filestore 20 --debug-ms 1'
> done
debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1
debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1
debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1
debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1
debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1
debug_osd=20/20 debug_filestore=20/20 debug_ms=1/1
# date
Mon Mar 7 16:03:38 CST 2016
# ceph pg deep-scrub 70.459
instructing pg 70.459 on osd.307 to deep-scrub
Scrub finished around
# date
Mon Mar 7 16:13:03 CST 2016
I've tar'd+gziped the files which can be downloaded from here. The logs start a minute or two after today at 16:00.
Oddly(to me anyways), this pg is now active+clean:
# ceph pg dump | grep 70.459
dumped all in format plain
70.459 21377 0 0 0 0 64515446306 3088 3088 active+clean 2016-03-07 16:26:57.796537 279563'212832 279602:628151 [307,210,273,191,132,450] 307 [307,210,273,191,132,450] 307 279563'212832 2016-03-07 16:12:30.741984 279563'212832 2016-03-07 16:12:30.741984
Regards,
Jeff
On Mon, Mar 7, 2016 at 4:11 PM, Samuel Just <sjust@xxxxxxxxxx> wrote:
I think the unfound object on repair is fixed by
d51806f5b330d5f112281fbb95ea6addf994324e (not in hammer yet). I
opened http://tracker.ceph.com/issues/15002 for the backport and to
make sure it's covered in ceph-qa-suite. No idea at this time why the
objects are disappearing though.
-Sam
On Mon, Mar 7, 2016 at 1:57 PM, Samuel Just <sjust@xxxxxxxxxx> wrote:
> The one just scrubbed and now inconsistent.
> -Sam
>
> On Mon, Mar 7, 2016 at 1:57 PM, Jeffrey McDonald <jmcdonal@xxxxxxx> wrote:
>> Do you want me to enable this for the pg already with unfound objects or the
>> placement group just scrubbed and now inconsistent?
>> Jeff
>>
>> On Mon, Mar 7, 2016 at 3:54 PM, Samuel Just <sjust@xxxxxxxxxx> wrote:
>>>
>>> Can you enable
>>>
>>> debug osd = 20
>>> debug filestore = 20
>>> debug ms = 1
>>>
>>> on all osds in that PG, rescrub, and convey to us the resulting logs?
>>> -Sam
>>>
>>> On Mon, Mar 7, 2016 at 1:36 PM, Jeffrey McDonald <jmcdonal@xxxxxxx> wrote:
>>> > Here is a PG which just went inconsistent:
>>> >
>>> > pg 70.459 is active+clean+inconsistent, acting [307,210,273,191,132,450]
>>> >
>>> > Attached is the result of a pg query on this. I will wait for your
>>> > feedback before issuing a repair.
>>> >
>>> > From what I read, the inconsistencies are more likely the result of ntp,
>>> > but
>>> > all nodes have the local ntp master and all are showing sync.
>>> >
>>> > Regards,
>>> > Jeff
>>> >
>>> > On Mon, Mar 7, 2016 at 3:15 PM, Gregory Farnum <gfarnum@xxxxxxxxxx>
>>> > wrote:
>>> >>
>>> >> [ Keeping this on the users list. ]
>>> >>
>>> >> Okay, so next time this happens you probably want to do a pg query on
>>> >> the PG which has been reported as dirty. I can't help much beyond
>>> >> that, but hopefully Kefu or David will chime in once there's a little
>>> >> more for them to look at.
>>> >> -Greg
>>> >>
>>> >> On Mon, Mar 7, 2016 at 1:00 PM, Jeffrey McDonald <jmcdonal@xxxxxxx>
>>> >> wrote:
>>> >> > Hi Greg,
>>> >> >
>>> >> > I'm running the ceph version hammer,
>>> >> > ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)
>>> >> >
>>> >> > The hardware migration was performed by just setting the crush map to
>>> >> > zero
>>> >> > for the OSD we wanted to retire. The system was performing poorly
>>> >> > with
>>> >> > these older OSDs and we had a difficult time maintaining stability of
>>> >> > the
>>> >> > system. The old OSDs are still there but all of the data is now
>>> >> > migrated
>>> >> > to new and/or existing hardware.
>>> >> >
>>> >> > Thanks,
>>> >> > Jeff
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Mon, Mar 7, 2016 at 2:56 PM, Gregory Farnum <gfarnum@xxxxxxxxxx>
>>> >> > wrote:
>>> >> >>
>>> >> >> On Mon, Mar 7, 2016 at 12:07 PM, Jeffrey McDonald <jmcdonal@xxxxxxx>
>>> >> >> wrote:
>>> >> >> > Hi,
>>> >> >> >
>>> >> >> > For a while, we've been seeing inconsistent placement groups on
>>> >> >> > our
>>> >> >> > erasure
>>> >> >> > coded system. The placement groups go from a state of
>>> >> >> > active+clean
>>> >> >> > to
>>> >> >> > active+clean+inconsistent after a deep scrub:
>>> >> >> >
>>> >> >> >
>>> >> >> > 2016-03-07 13:45:42.044131 7f385d118700 -1 log_channel(cluster)
>>> >> >> > log
>>> >> >> > [ERR] :
>>> >> >> > 70.320s0 deep-scrub stat mismatch, got 21446/21428 objects, 0/0
>>> >> >> > clones,
>>> >> >> > 21446/21428 dirty, 0/0 omap, 0/0 hit_set_archive, 0/0 whiteouts,
>>> >> >> > 64682334170/64624353083 bytes,0/0 hit_set_archive bytes.
>>> >> >> > 2016-03-07 13:45:42.044416 7f385d118700 -1 log_channel(cluster)
>>> >> >> > log
>>> >> >> > [ERR] :
>>> >> >> > 70.320s0 deep-scrub 18 missing, 0 inconsistent objects
>>> >> >> > 2016-03-07 13:45:42.044464 7f385d118700 -1 log_channel(cluster)
>>> >> >> > log
>>> >> >> > [ERR] :
>>> >> >> > 70.320 deep-scrub 73 errors
>>> >> >> >
>>> >> >> > So I tell the placement group to perform a repair:
>>> >> >> >
>>> >> >> > 2016-03-07 13:49:26.047177 7f385d118700 0 log_channel(cluster)
>>> >> >> > log
>>> >> >> > [INF] :
>>> >> >> > 70.320 repair starts
>>> >> >> > 2016-03-07 13:49:57.087291 7f3858b0a700 0 -- 10.31.0.2:6874/13937
>>> >> >> > >>
>>> >> >> > 10.31.0.6:6824/8127 pipe(0x2e578000 sd=697 :6874
>>> >> >> >
>>> >> >> > The repair finds missing shards and repairs them, but then I have
>>> >> >> > 18
>>> >> >> > 'unfound objects' :
>>> >> >> >
>>> >> >> >
>>> >> >> > 2016-03-07 13:51:28.467590 7f385d118700 -1 log_channel(cluster)
>>> >> >> > log
>>> >> >> > [ERR] :
>>> >> >> > 70.320s0 repair stat mismatch, got 21446/21428 objects, 0/0
>>> >> >> > clones,
>>> >> >> > 21446/21428 dirty, 0/0 omap, 0/0 hit_set_archive, 0/0 whiteouts,
>>> >> >> > 64682334170/64624353083 bytes,0/0 hit_set_archive bytes.
>>> >> >> > 2016-03-07 13:51:28.468358 7f385d118700 -1 log_channel(cluster)
>>> >> >> > log
>>> >> >> > [ERR] :
>>> >> >> > 70.320s0 repair 18 missing, 0 inconsistent objects
>>> >> >> > 2016-03-07 13:51:28.469431 7f385d118700 -1 log_channel(cluster)
>>> >> >> > log
>>> >> >> > [ERR] :
>>> >> >> > 70.320 repair 73 errors, 73 fixed
>>> >> >> >
>>> >> >> >
>>> >> >> > I've traced one of the unfound objects all the way through the
>>> >> >> > system
>>> >> >> > and
>>> >> >> > I've found that they are not really lost. I can fail over the
>>> >> >> > osd
>>> >> >> > and
>>> >> >> > recover the files. This is happening quite regularly now after a
>>> >> >> > large
>>> >> >> > migration of data from old hardware to new(migration is now
>>> >> >> > complete).
>>> >> >> >
>>> >> >> > The system sets the PG into 'recovery', but we've seen the system
>>> >> >> > in
>>> >> >> > a
>>> >> >> > recovering state for many days. Should we just be patient or do
>>> >> >> > we
>>> >> >> > need
>>> >> >> > to dig further into the issue?
>>> >> >>
>>> >> >> You may need to dig into this more, although I'm not sure what the
>>> >> >> issue is likely to be. What version of Ceph are you running? How did
>>> >> >> you do this hardware migration?
>>> >> >> -Greg
>>> >> >>
>>> >> >> >
>>> >> >> >
>>> >> >> > pg 70.320 is stuck unclean for 704.803040, current state
>>> >> >> > active+recovering,
>>> >> >> > last acting [277,101,218,49,304,412]
>>> >> >> > pg 70.320 is active+recovering, acting [277,101,218,49,304,412],
>>> >> >> > 18
>>> >> >> > unfound
>>> >> >> >
>>> >> >> > There is no indication of any problems with down OSDs or network
>>> >> >> > issues
>>> >> >> > with
>>> >> >> > OSDs.
>>> >> >> >
>>> >> >> > Thanks,
>>> >> >> > Jeff
>>> >> >> >
>>> >> >> >
>>> >> >> > --
>>> >> >> >
>>> >> >> > Jeffrey McDonald, PhD
>>> >> >> > Assistant Director for HPC Operations
>>> >> >> > Minnesota Supercomputing Institute
>>> >> >> > University of Minnesota Twin Cities
>>> >> >> > 599 Walter Library email: jeffrey.mcdonald@xxxxxxxxxxx
>>> >> >> > 117 Pleasant St SE phone: +1 612 625-6905
>>> >> >> > Minneapolis, MN 55455 fax: +1 612 624-8861
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > _______________________________________________
>>> >> >> > ceph-users mailing list
>>> >> >> > ceph-users@xxxxxxxxxxxxxx
>>> >> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> >> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> >
>>> >> > Jeffrey McDonald, PhD
>>> >> > Assistant Director for HPC Operations
>>> >> > Minnesota Supercomputing Institute
>>> >> > University of Minnesota Twin Cities
>>> >> > 599 Walter Library email: jeffrey.mcdonald@xxxxxxxxxxx
>>> >> > 117 Pleasant St SE phone: +1 612 625-6905
>>> >> > Minneapolis, MN 55455 fax: +1 612 624-8861
>>> >> >
>>> >> >
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> >
>>> > Jeffrey McDonald, PhD
>>> > Assistant Director for HPC Operations
>>> > Minnesota Supercomputing Institute
>>> > University of Minnesota Twin Cities
>>> > 599 Walter Library email: jeffrey.mcdonald@xxxxxxxxxxx
>>> > 117 Pleasant St SE phone: +1 612 625-6905
>>> > Minneapolis, MN 55455 fax: +1 612 624-8861
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > ceph-users mailing list
>>> > ceph-users@xxxxxxxxxxxxxx
>>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> >
>>
>>
>>
>>
>> --
>>
>> Jeffrey McDonald, PhD
>> Assistant Director for HPC Operations
>> Minnesota Supercomputing Institute
>> University of Minnesota Twin Cities
>> 599 Walter Library email: jeffrey.mcdonald@xxxxxxxxxxx
>> 117 Pleasant St SE phone: +1 612 625-6905
>> Minneapolis, MN 55455 fax: +1 612 624-8861
>>
>>
Jeffrey McDonald, PhD Assistant Director for HPC Operations Minnesota Supercomputing Institute University of Minnesota Twin Cities 599 Walter Library email: jeffrey.mcdonald@xxxxxxxxxxx 117 Pleasant St SE phone: +1 612 625-6905 Minneapolis, MN 55455 fax: +1 612 624-8861
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
