On Sat, Dec 27, 2014 at 4:09 PM, Andrey Korolyov <andrey@xxxxxxx> wrote:
> On Tue, Dec 23, 2014 at 4:17 AM, Samuel Just <sam.just@xxxxxxxxxxx> wrote:
>> Oh, that's a bit less interesting. The bug might be still around though.
>> -Sam
>>
>> On Mon, Dec 22, 2014 at 2:50 PM, Andrey Korolyov <andrey@xxxxxxx> wrote:
>>> On Tue, Dec 23, 2014 at 1:12 AM, Samuel Just <sam.just@xxxxxxxxxxx> wrote:
>>>> You'll have to reproduce with logs on all three nodes. I suggest you
>>>> open a high priority bug and attach the logs.
>>>>
>>>> debug osd = 20
>>>> debug filestore = 20
>>>> debug ms = 1
>>>>
>>>> I'll be out for the holidays, but I should be able to look at it when
>>>> I get back.
>>>> -Sam
>>>>
>>>
>>>
>>> Thanks Sam,
>>>
>>> although I am not sure if it makes not only a historical interest (the
>>> mentioned cluster running cuttlefish), I`ll try to collect logs for
>>> scrub.
>
> Same stuff:
> https://www.mail-archive.com/ceph-users@xxxxxxxxxxxxxx/msg15447.html
> https://www.mail-archive.com/ceph-users@xxxxxxxxxxxxxx/msg14918.html
>
> Looks like issue is still with us, though it requires meta or file
> structure corruption to show itself. I`ll check if it can be
> reproduced via rsync -X sec pg subdir -> pri pg subdir or vice-versa.
> Mine case shows slightly different pathnames for same objects with
> same checksums, may be a root reason then. As every case mentioned,
> including mine, happened in oh-shit-hardware-is-broken case, I suggest
> that the incurable corruption happens during primary backfill from
> active replica at the recovery time.
Recovery/backfill from corrupted primary copy results to crash
(attached) of primary OSD, for example it can be triggered by purging
one of secondary copies (top of cuttlefish branch for line numbers).
Although as secondaries preserve same data with same checksums, it is
possible to destroy both meta record and pg directory and refill
primary back. The interesting point is that the corrupted primary was
completely refilled after hardware failure, but looks like it survived
long enough after a failure event to spread corruption to the copies,
I simply can not imagine better explanation.
Thread 1 (Thread 0x7f193190d700 (LWP 64087)):
#0 0x00007f194a47ab7b in raise () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x0000000000857d59 in reraise_fatal (signum=6)
at global/signal_handler.cc:58
#2 handle_fatal_signal (signum=6) at global/signal_handler.cc:104
#3 <signal handler called>
#4 0x00007f1948879405 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007f194887cb5b in abort () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x00007f194917789d in __gnu_cxx::__verbose_terminate_handler() ()
from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x00007f1949175996 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8 0x00007f19491759c3 in std::terminate() ()
from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#9 0x00007f1949175bee in __cxa_throw ()
from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x000000000090436a in ceph::__ceph_assert_fail (
assertion=0x9caf67 "r >= 0", file=<optimized out>, line=7115,
func=0x9d1900 "void ReplicatedPG::scan_range(hobject_t, int, int, PG::BackfillInterval*)") at common/assert.cc:77
#11 0x000000000065de69 in ReplicatedPG::scan_range (this=this@entry=0x4df6000,
begin=..., min=min@entry=32, max=max@entry=64, bi=bi@entry=0x4df6d40)
at osd/ReplicatedPG.cc:7115
#12 0x000000000066f5c6 in ReplicatedPG::recover_backfill (
this=this@entry=0x4df6000, max=max@entry=1) at osd/ReplicatedPG.cc:6923
#13 0x000000000067c18d in ReplicatedPG::start_recovery_ops (this=0x4df6000,
max=1, prctx=<optimized out>) at osd/ReplicatedPG.cc:6561
#14 0x00000000006f2340 in OSD::do_recovery (this=0x2ba7000, pg=pg@entry=
0x4df6000) at osd/OSD.cc:6104
#15 0x0000000000735361 in OSD::RecoveryWQ::_process (this=<optimized out>,
pg=0x4df6000) at osd/OSD.h:1248
#16 0x00000000008faeba in ThreadPool::worker (this=0x2ba75e0, wt=0x7be1540)
at common/WorkQueue.cc:119
#17 0x00000000008fc160 in ThreadPool::WorkThread::entry (this=<optimized out>)
at common/WorkQueue.h:316
#18 0x00007f194a472e9a in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#19 0x00007f19489353dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#20 0x0000000000000000 in ?? ()
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
