On 10/17/2012 11:23 AM, Nick Couchman wrote:
Hmmm...I don't seem to have the dbg packages built...will have to go back and figure out how to build those.
Ah I thought you had installed from debian binaries. If you compiled
ceph yourself, to get the debugging symbols you have to reconfigure with
-g on CXXFLAGS:
./configure CXXFLAGS=-g
-sam
-Nick
On 2012/10/17 at 09:53, Sam Lang <sam.lang@xxxxxxxxxxx> wrote:
On 10/17/2012 09:42 AM, Nick Couchman wrote:
Thanks...here's the backtrace:
(gdb) bt
#0 0x00000000004dcfea in ESession::replay(MDS*) ()
#1 0x00000000006a2446 in MDLog::_replay_thread() ()
#2 0x00000000004cf5ed in MDLog::ReplayThread::entry() ()
#3 0x00007ffff764df05 in start_thread () from /lib64/libpthread.so.0
#4 0x00007ffff680d10d in clone () from /lib64/libc.so.6
Hi Nick,
This doesn't have the debug symbols (line numbers in the source) we were
hoping for. Could you install the ceph-dpg package and rerun? You will
probably have to first uninstall the ceph package.
Thanks,
-sam
On 2012/10/17 at 07:34, Sam Lang <sam.lang@xxxxxxxxxxx> wrote:
On 10/16/2012 06:04 PM, Gregory Farnum wrote:
Okay, that's the right debugging but it wasn't quite as helpful on its
own as I expected. Can you get a core dump (you might already have
one, depending on system settings) of the crash and open it up with
gdb and get a full backtrace?
You can also run the mds directly in gdb and avoid any core file ulimit
settings you have set:
> gdb --args ceph-mds -n mds.b -c /etc/ceph/ceph.conf -f
...
(gdb) run
Once you hit the segfault you can get the backtrace with:
(gdb) bt
-sam
-Greg
On Mon, Oct 15, 2012 at 10:59 AM, Nick Couchman <Nick.Couchman@xxxxxxxxx>
wrote:
Well, hopefully this is still okay...8.5MB bzip2d, 230MB unzipped.
-Nick
On 2012/10/15 at 11:47, Gregory Farnum <greg@xxxxxxxxxxx> wrote:
Yeah, zip it and post * somebody's going to have to download it and
do
fun things. :)
-Greg
On Mon, Oct 15, 2012 at 10:43 AM, Nick Couchman
<Nick.Couchman@xxxxxxxxx>
wrote:
Anywhere in particular I should make it available? It's a little
over a
million lines of debug in the file - I can put it on a pastebin, if
that
works, or perhaps zip it up and throw it somewhere?
-Nick
On 2012/10/15 at 11:26, Gregory Farnum <greg@xxxxxxxxxxx> wrote:
Something in the MDS log is bad or is poking at a bug in the code.
Can
you turn on MDS debugging and restart a daemon and put that log
somewhere accessible?
debug mds = 20
debug journaler = 20
debug ms = 1
-Greg
On Mon, Oct 15, 2012 at 10:02 AM, Nick Couchman
<Nick.Couchman@xxxxxxxxx>
wrote:
Well, both of my MDSs seem to be down right now, and then
continually
segfault (every time I try to start them) with the following:
ceph-mdsmon-a:~ # ceph-mds -n mds.b -c /etc/ceph/ceph.conf -f
starting mds.b at :/0
*** Caught signal (Segmentation fault) **
in thread 7fbe0d61d700
ceph version 0.48.1argonaut
(commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c)
1: ceph-mds() [0x7ef83a]
2: (()+0xfd00) [0x7fbe15a0cd00]
3: (ESession::replay(MDS*)+0x3ea) [0x4dcfea]
4: (MDLog::_replay_thread()+0x6b6) [0x6a2446]
5: (MDLog::ReplayThread::entry()+0xd) [0x4cf5ed]
6: (()+0x7f05) [0x7fbe15a04f05]
7: (clone()+0x6d) [0x7fbe14bc410d]
2012-10-15 10:57:35.449161 7fbe0d61d700 -1 *** Caught signal
(Segmentation
fault) **
in thread 7fbe0d61d700
ceph version 0.48.1argonaut
(commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c)
1: ceph-mds() [0x7ef83a]
2: (()+0xfd00) [0x7fbe15a0cd00]
3: (ESession::replay(MDS*)+0x3ea) [0x4dcfea]
4: (MDLog::_replay_thread()+0x6b6) [0x6a2446]
5: (MDLog::ReplayThread::entry()+0xd) [0x4cf5ed]
6: (()+0x7f05) [0x7fbe15a04f05]
7: (clone()+0x6d) [0x7fbe14bc410d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to
interpret this.
0> 2012-10-15 10:57:35.449161 7fbe0d61d700 -1 *** Caught
signal
(Segmentation fault) **
in thread 7fbe0d61d700
ceph version 0.48.1argonaut
(commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c)
1: ceph-mds() [0x7ef83a]
2: (()+0xfd00) [0x7fbe15a0cd00]
3: (ESession::replay(MDS*)+0x3ea) [0x4dcfea]
4: (MDLog::_replay_thread()+0x6b6) [0x6a2446]
5: (MDLog::ReplayThread::entry()+0xd) [0x4cf5ed]
6: (()+0x7f05) [0x7fbe15a04f05]
7: (clone()+0x6d) [0x7fbe14bc410d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to
interpret this.
Segmentation fault
Anyone have any hints on recovering? I'm running 0.48.1argonaut -
I can
attempt to upgrade to 0.48.2 and see if that helps, but I figured
if anyone
can offer any insight as to what to do to get the replay to run
without
segfaulting?
--------
This e-mail may contain confidential and privileged material for
the sole use
of the intended recipient. If this email is not intended for you,
or you
are
not responsible for the delivery of this message to the intended
recipient,
please note that this message may contain SEAKR Engineering
(SEAKR)
Privileged/Proprietary Information. In such a case, you are
strictly
prohibited from downloading, photocopying, distributing or
otherwise using
this message, its contents or attachments in any way. If you have
received
this message in error, please notify us immediately by replying to
this
e-mail
and delete the message from your mailbox. Information contained in
this
message that does not relate to the business of SEAKR is neither
endorsed by
nor attributable to SEAKR.
--
To unsubscribe from this list: send the line "unsubscribe
ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
http://vger.kernel.org/majordomo-info.html
--------
This e-mail may contain confidential and privileged material for the
sole use
of the intended recipient. If this email is not intended for you, or
you are
not responsible for the delivery of this message to the intended
recipient,
please note that this message may contain SEAKR Engineering (SEAKR)
Privileged/Proprietary Information. In such a case, you are strictly
prohibited from downloading, photocopying, distributing or otherwise
using
this message, its contents or attachments in any way. If you have
received
this message in error, please notify us immediately by replying to
this e-mail
and delete the message from your mailbox. Information contained in
this
message that does not relate to the business of SEAKR is neither
endorsed by
nor attributable to SEAKR.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--------
This e-mail may contain confidential and privileged material for the sole use
of the intended recipient. If this email is not intended for you, or you
are
not responsible for the delivery of this message to the intended recipient,
please note that this message may contain SEAKR Engineering (SEAKR)
Privileged/Proprietary Information. In such a case, you are strictly
prohibited from downloading, photocopying, distributing or otherwise using
this message, its contents or attachments in any way. If you have received
this message in error, please notify us immediately by replying to this
e-mail
and delete the message from your mailbox. Information contained in this
message that does not relate to the business of SEAKR is neither endorsed by
nor attributable to SEAKR.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--------
This e-mail may contain confidential and privileged material for the sole use
of the intended recipient. If this email is not intended for you, or you are
not responsible for the delivery of this message to the intended recipient,
please note that this message may contain SEAKR Engineering (SEAKR)
Privileged/Proprietary Information. In such a case, you are strictly
prohibited from downloading, photocopying, distributing or otherwise using
this message, its contents or attachments in any way. If you have received
this message in error, please notify us immediately by replying to this e-mail
and delete the message from your mailbox. Information contained in this
message that does not relate to the business of SEAKR is neither endorsed by
nor attributable to SEAKR.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--------
This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
