I'm seeing weird messages in my monitor logs that don't correlate to admin activity: 2013-04-07 22:54:11.528871 7f2e9e6c8700 1 -- [2001:<something>::20]:6789/0 --> [2001:<something>::20]:0/1920 -- mon_command_ack([auth,get-or-create,client.admin,mon,allow *,osd,allow *,mds,allow]=-13 access denied v134192) v1 -- ?+0 0x37bfc00 con 0x3716840 It's also writing out a bunch of empty files along the lines of "ceph.client.admin.keyring.1008.tmp" in /etc/ceph/ Could this be related to the mon trying to "Starting ceph-create-keys" when starting? This could be the cause of, or just associated with, some general instability of the monitor cluster. After increasing the logging level I did catch one crash: ceph version 0.60 (f26f7a39021dbf440c28d6375222e21c94fe8e5c) 1: /usr/bin/ceph-mon() [0x5834fa] 2: (()+0xfcb0) [0x7f4b03328cb0] 3: (gsignal()+0x35) [0x7f4b01efe425] 4: (abort()+0x17b) [0x7f4b01f01b8b] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f4b0285069d] 6: (()+0xb5846) [0x7f4b0284e846] 7: (()+0xb5873) [0x7f4b0284e873] 8: (()+0xb596e) [0x7f4b0284e96e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1df) [0x636c8f] 10: (PaxosService::propose_pending()+0x46d) [0x4dee3d] 11: (MDSMonitor::tick()+0x1c62) [0x51cdd2] 12: (MDSMonitor::on_active()+0x1a) [0x512ada] 13: (PaxosService::_active()+0x31d) [0x4e067d] 14: (Context::complete(int)+0xa) [0x4b7b4a] 15: (finish_contexts(CephContext*, std::list<Context*, std::allocator<Context*> >&, int)+0x95) [0x4ba5a5] 16: (Paxos::handle_last(MMonPaxos*)+0xbef) [0x4da92f] 17: (Paxos::dispatch(PaxosServiceMessage*)+0x26b) [0x4dad8b] 18: (Monitor::_ms_dispatch(Message*)+0x149f) [0x4b310f] 19: (Monitor::ms_dispatch(Message*)+0x32) [0x4c9d12] 20: (DispatchQueue::entry()+0x341) [0x698da1] 21: (DispatchQueue::DispatchThread::entry()+0xd) [0x626c5d] 22: (()+0x7e9a) [0x7f4b03320e9a] 23: (clone()+0x6d) [0x7f4b01fbbcbd] The complete log is at: http://goo.gl/UmNs3 Does anyone recognize what's going on? -- Matthew _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |