Hi Wido, On Thu, 6 May 2010, Wido den Hollander wrote: > Today i started to see new crashes on my cluster, this time my MDS > (both) are crashing with a different message. > > What did i do? > > 1. Unpack a tarball with a lot of small files > 2. Chown and rename some files > 3. Remove the entire directory > > The removal of the files didn't go as planned, some directories have > become unable to remove, the message is that the directory is not empty. > > Afterwards i see one of my MDS (seems a random order) to use up 100% and > also 100% disk (I thought the MDS didn't use any disk I/O?) The only IO the mds is doing is logging. It sounds like that's where the problem is. > The lines are almost the same at both hosts, they seem to shutdown? They do this if the monitor tells them they've been replaced. Which happens if they're slow/laggy enough that they can't exchange timely heartbeat messages. Probably due to them getting hung up in some bad loop in the logging code... > Loaded symbols for /lib/libz.so.1 > Core was generated by `/usr/bin/cmds -i 0 -c /etc/ceph/ceph.conf'. > Program terminated with signal 11, Segmentation fault. > #0 0x0000000000687d48 in Logger::_flush(bool) () > (gdb) bt > #0 0x0000000000687d48 in Logger::_flush(bool) () > #1 0x000000000068894e in ?? () > #2 0x000000000068c839 in SafeTimer::EventWrapper::finish(int) () > #3 0x000000000068e1d4 in Timer::timer_entry() () > #4 0x0000000000477b2d in Timer::TimerThread::entry() () > #5 0x0000000000488b3a in Thread::_entry_func(void*) () > #6 0x00007f8951a2fa04 in start_thread () from /lib/libpthread.so.0 > #7 0x00007f8950c6780d in clone () from /lib/libc.so.6 > #8 0x0000000000000000 in ?? () ...here? If you install the ceph-dbg package, and run gdb from a ceph source directory using the debug version of the binary, gdb should give you line numbers and such: $ cd ceph/src $ gdb /usr/lib/debug/usr/bin/cmds /path/to/core > Core was generated by `/usr/bin/cmds -i 1 -c /etc/ceph/ceph.conf'. > Program terminated with signal 11, Segmentation fault. > #0 0x00000000005c0b47 in CDir::_fetched(ceph::buffer::list&) () > (gdb) bt > #0 0x00000000005c0b47 in CDir::_fetched(ceph::buffer::list&) () > #1 0x000000000062f5a5 in Objecter::handle_osd_op_reply(MOSDOpReply*) () > #2 0x00000000004a0e9d in MDS::_dispatch(Message*) () > #3 0x00000000004a0f97 in MDS::ms_dispatch(Message*) () > #4 0x000000000047ef29 in SimpleMessenger::dispatch_entry() () > #5 0x0000000000477d7c in SimpleMessenger::DispatchThread::entry() () > #6 0x0000000000488b3a in Thread::_entry_func(void*) () > #7 0x00007f235278ba04 in start_thread () from /lib/libpthread.so.0 > #8 0x00007f23519c380d in clone () from /lib/libc.so.6 > #9 0x0000000000000000 in ?? () > (gdb) This is something else. Can you try to get a line number for this too (using the debug binary)? > Right now my MDS are running in a KVM Virtual Machine, both on the same > physical machine, but that shouldn't be a problem, should it? The > performance could be a lot lower, but the crashes shouldn't be there. It shouldn't matter. Especially for the mds. (The osd might have problems due to things like fsync not being honest using qcow2 images, etc.) Thanks- sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
