I've managed to get joao's assistance in tracking down the issue. I'll be updating the bug 7210.
Thanks joao and all!
- WP
On Thu, Mar 6, 2014 at 6:25 PM, YIP Wai Peng <yipwp@xxxxxxxxxxxxxxx> wrote:
Ok, I think I got bitten by http://tracker.ceph.com/issues/7210, or rather, the cppool command in http://www.sebastien-han.fr/blog/2013/03/12/ceph-change-pg-number-on-the-fly/I did use "rados cppool <oldpool> <newpool>" in a pool with snapshots (openstack glance). A user feedback that ceph crashed when he deleted an image in openstack.I'm now wondering if I can ignore the operation, or the openstack glance pool, and get the mons to start up again. Any help will be greatly appreciated!- WPOn Thu, Mar 6, 2014 at 5:33 PM, YIP Wai Peng <yipwp@xxxxxxxxxxxxxxx> wrote:Hi,I am currently facing a horrible situation. All my mons are crashing on startup.Here's a dump of mon.a.log. The last few ops are below. It seems to crash trying to remove a snap? Any ideas?- WP<snip>-10> 2014-03-06 17:04:38.838490 7fb2a541a700 1 -- 192.168.116.24:6789/0 --> osd.9 192.168.116.27:6955/11604 -- mon_subscribe_ack(300s) v1 -- ?+0 0x32d0540-9> 2014-03-06 17:04:38.838511 7fb2a541a700 1 -- 192.168.116.24:6789/0 <== osd.1 192.168.116.24:6812/30221 6 ==== mon_subscribe({monmap=14+,osd_pg_creates=0}) v2 ==== 50+0+0 (3009623588 0 0) 0x32d71c0 con 0x3224200-8> 2014-03-06 17:04:38.838527 7fb2a541a700 1 -- 192.168.116.24:6789/0 --> osd.1 192.168.116.24:6812/30221 -- mon_subscribe_ack(300s) v1 -- ?+0 0x32d3640-7> 2014-03-06 17:04:38.838545 7fb2a541a700 1 -- 192.168.116.24:6789/0 <== mon.2 192.168.116.26:6789/0 1868286886 ==== forward(pg_stats(0 pgs tid 790 v 0) v1 caps allow rwx) to leader v1 ==== 398+0+0 (2470115819 0 0) 0x32f8c80 con 0x2f91760-6> 2014-03-06 17:04:38.838570 7fb2a541a700 1 mon.a@0(leader).paxos(paxos active c 7361285..7361906) is_readable now=2014-03-06 17:04:38.838571 lease_expire=2014-03-06 17:04:43.820707 has v0 lc 7361906-5> 2014-03-06 17:04:38.838595 7fb2a541a700 1 mon.a@0(leader).paxos(paxos active c 7361285..7361906) is_readable now=2014-03-06 17:04:38.838597 lease_expire=2014-03-06 17:04:43.820707 has v0 lc 7361906-4> 2014-03-06 17:04:38.838626 7fb2a541a700 1 -- 192.168.116.24:6789/0 <== mon.2 192.168.116.26:6789/0 1868286887 ==== forward(osd_pgtemp(e26089 {6.fe=[]} v26089) v1 caps allow rwx) to leader v1 ==== 297+0+0 (2442013554 0 0) 0x32f8780 con 0x2f91760-3> 2014-03-06 17:04:38.838662 7fb2a541a700 1 mon.a@0(leader).paxos(paxos active c 7361285..7361906) is_readable now=2014-03-06 17:04:38.838665 lease_expire=2014-03-06 17:04:43.820707 has v0 lc 7361906-2> 2014-03-06 17:04:38.838696 7fb2a541a700 1 -- 192.168.116.24:6789/0 <== mon.2 192.168.116.26:6789/0 1868286888 ==== forward(pool_op(delete unmanaged snap pool 10 auid 0 tid 27 name v0) v4 caps allow r) to leader v1 ==== 313+0+0 (3176715156 0 0) 0x32f8500 con 0x2f91760-1> 2014-03-06 17:04:38.838715 7fb2a541a700 1 mon.a@0(leader).paxos(paxos active c 7361285..7361906) is_readable now=2014-03-06 17:04:38.838717 lease_expire=2014-03-06 17:04:43.820707 has v0 lc 73619060> 2014-03-06 17:04:38.840833 7fb2a541a700 -1 osd/osd_types.cc: In function 'void pg_pool_t::remove_unmanaged_snap(snapid_t)' thread 7fb2a541a700 time 2014-03-06 17:04:38.838745osd/osd_types.cc: 799: FAILED assert(is_unmanaged_snaps_mode())ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60)1: /usr/bin/ceph-mon() [0x6c96e9]2: (OSDMonitor::prepare_pool_op(MPoolOp*)+0x970) [0x5c3ad0]3: (OSDMonitor::prepare_update(PaxosServiceMessage*)+0x1ab) [0x5c3d8b]4: (PaxosService::dispatch(PaxosServiceMessage*)+0xa1a) [0x5940ea]5: (Monitor::dispatch(MonSession*, Message*, bool)+0xdb) [0x56320b]6: (Monitor::_ms_dispatch(Message*)+0x1fb) [0x5639fb]7: (Monitor::handle_forward(MForward*)+0x9c2) [0x565092]8: (Monitor::dispatch(MonSession*, Message*, bool)+0x400) [0x563530]9: (Monitor::_ms_dispatch(Message*)+0x1fb) [0x5639fb]10: (Monitor::ms_dispatch(Message*)+0x32) [0x57f212]11: (DispatchQueue::entry()+0x582) [0x7de6c2]12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7d994d]13: (()+0x79d1) [0x7fb2ac05e9d1]14: (clone()+0x6d) [0x7fb2aad95b6d]NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.<snip>
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
