Re: Adding a new OSD crash the monitors (assertion failure)

[Guang <yguang11@xxxxxxxxx>]

I just found the trick..

When I am using a default crush, which use straw bucket type, things are good.

However, for the error I posted below, it is using tree bucket type.

Is it related?

Thanks,

Guang

On Oct 30, 2013, at 6:52 PM, Guang wrote:

Hi all,

Today I tried to add a new OSD into the cluster and immediately it get the monitors crashed.

Platform: RHEL6.4

Steps to add new monitor:

  1. sudo ceph-disk zap /dev/sdh

  2. sudo ceph-disk activate /dev/sdh

Then the monitor got crashed with the following logs:

013-10-30 02:17:14.252726 7f44395a9700  0 mon.ceph2@0(leader) e2 handle_command mon_command({"prefix": "osd crush create-or-move", "args": ["root=default", "host=ceph8"], "id": 24, "weight": 0.40999999999999998} v 0) v1

2013-10-30 02:17:14.252792 7f44395a9700  1 mon.ceph2@0(leader).paxos(paxos active c 322285..323030) is_readable now=2013-10-30 02:17:14.252794 lease_expire=2013-10-30 02:17:19.063672 has v0 lc 323030

2013-10-30 02:17:14.252911 7f44395a9700  0 mon.ceph2@0(leader).osd e916 create-or-move crush item name 'osd.24' initial_weight 0.41 at location {host=ceph8,root=default}

2013-10-30 02:17:14.255347 7f44395a9700 -1 crush/CrushWrapper.cc: In function 'int CrushWrapper::insert_item(CephContext*, int, float, std::string, const std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<const std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&)' thread 7f44395a9700 time 2013-10-30 02:17:14.253030

crush/CrushWrapper.cc: 413: FAILED assert(!r)

 ceph version 0.67.3 (408cd61584c72c0d97b774b3d8f95c6b1b06341a)

 1: (CrushWrapper::insert_item(CephContext*, int, float, std::string, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&)+0x14b4) [0x6b9514]

 2: (CrushWrapper::create_or_move_item(CephContext*, int, float, std::string, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&)+0x2d6) [0x6ba0f6]

 3: (OSDMonitor::prepare_command(MMonCommand*)+0x150a) [0x5aa89a]

 4: (OSDMonitor::prepare_update(PaxosServiceMessage*)+0x20b) [0x5b2e2b]

 5: (PaxosService::dispatch(PaxosServiceMessage*)+0xa20) [0x58bea0]

 6: (Monitor::handle_command(MMonCommand*)+0xdec) [0x557ddc]

 7: (Monitor::_ms_dispatch(Message*)+0xc2f) [0x5600af]

 8: (Monitor::handle_forward(MForward*)+0x990) [0x55f0c0]

 9: (Monitor::_ms_dispatch(Message*)+0xd53) [0x5601d3]

 10: (Monitor::ms_dispatch(Message*)+0x32) [0x578742]

 11: (DispatchQueue::entry()+0x5a2) [0x7bdcc2]

 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7b690d]

 13: /lib64/libpthread.so.0() [0x3208a07851]

 14: (clone()+0x6d) [0x32086e890d]

 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Anyone else came across the same issue? Or am I missing anything when add a new OSD?

Thanks,

Guang

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com