Sage Weil <sage <at> inktank.com> writes: > > Hi Amit, > > > > common/Mutex.cc: In function 'void Mutex::Lock(bool)' thread 7f615f275700 > > time 2014-04-04 09:03:22.128731 > > common/Mutex.cc: 93: FAILED assert(r == 0) > > ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7) > > 1: (Mutex::Lock(bool)+0x1d3) [0x7f61576ac763] > > 2: (librados::IoCtxImpl::operate_read(object_t const&, ObjectOperation*, > > ceph::buffer::list*)+0x17b) [0x7f615765069b] > > This Mutex assertion usually triggers in use-after-free cases where the > pthread mutex id is invalid (because it has been deallocated). My guess > is that your IoCtx has been freed, or the shutdown() method has been > called on the cluster handle... is that possible? (Obviously not with the > code fragment above, but I'm guess that isn't a straight copy+paste from > your code?) > > sage > Thanks Sage for your input. I would look into the code to figure out if they are any calls to IoCtx::close() or Rados::shutdown(). However, on closer inspection I find lot of pthread_create() failing (refer log below) with error 11 (EAGAIN) indicating "insufficient resources to create another thread, or a system-imposed limit on the number of threads was encountered". Could Mutex assertion failure be a side effect of this? ------------------------------- Thread::try_create(): pthread_create failed with error 11 common/Thread.cc: In function 'void Thread::create(size_t)' thread 7f615f275700 time 2014-04-04 08:57:03.511877 common/Thread.cc: 110: FAILED assert(ret == 0) ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7) 1: (()+0x36004f) [0x7f615783804f] 2: (SimpleMessenger::connect_rank(entity_addr_t const&, int, Connection*, Message*)+0x17b) [0x7f61577c9b9b] 3: (SimpleMessenger::get_connection(entity_inst_t const&)+0x244) [0x7f61577ce824] 4: (Objecter::get_session(int)+0x1d1) [0x7f615765b391] 5: (Objecter::recalc_op_target(Objecter::Op*)+0x336) [0x7f615765c0d6] 6: (Objecter::_op_submit(Objecter::Op*)+0x43) [0x7f6157665993] 7: (librados::IoCtxImpl::operate_read(object_t const&, ObjectOperation*, ceph::buffer::list*)+0x2dc) [0x7f61576507fc] 8: (librados::IoCtxImpl::stat(object_t const&, unsigned long*, long*)+0x185) [0x7f6157653b05] 9: (librados::IoCtx::stat(std::string const&, unsigned long*, long*)+0x58) [0x7f6157628498] Regards, Amit Tiwary -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
