Hi,
i uploaded my ceph mon store to cephdrop
/home/cephdrop/ceph-mon-failed-assert-0.61.6/mon.tar.gz.
So hopefully someone can find the culprit soon.
It fails in OSDMonitor.cc here:
// if we trigger this, then there's something else going with the store
// state, and we shouldn't want to work around it without knowing what
// exactly happened.
assert(latest_full > 0);
Stefan
Am 24.07.2013 09:05, schrieb Stefan Priebe - Profihost AG:
> Hi,
>
> today i wanted to upgrade from 0.61.5 to 0.61.6 to get rid of the mon bug.
>
> But this ended in a complete desaster.
>
> What i've done:
> 1.) recompiled ceph tagged with 0.61.6
> 2.) installed new ceph version on all machines
> 3.) JUST tried to restart ONE mon
>
> this failed with:
> [1774]: (33) Numerical argument out of domain
> failed: 'ulimit -n 8192; /usr/bin/ceph-mon -i a --pid-file
> /var/run/ceph/mon.a.pid -c /etc/ceph/ceph.conf '
>
> 2013-07-24 08:41:43.086951 7f53c185d700 -1 mon.a@0(leader) e1 *** Got
> Signal Terminated ***
> 2013-07-24 08:41:43.088090 7f53c185d700 0 quorum service shutdown
> 2013-07-24 08:41:43.088094 7f53c185d700 0 mon.a@0(???).health(3840)
> HealthMonitor::service_shutdown 1 services
> 2013-07-24 08:41:43.088097 7f53c185d700 0 quorum service shutdown
> 2013-07-24 08:41:44.224104 7fae6384a780 0 ceph version
> 0.61.6-15-g85db066 (85db0667307ac803c753d16fa374dd2fc29d76f3), process
> ceph-mon, pid 29871
> 2013-07-24 08:41:56.097385 7fae6384a780 -1 mon/OSDMonitor.cc: In
> function 'virtual void OSDMonitor::update_from_paxos(bool*)' thread
> 7fae6384a780 time 2013-07-24 08:41:56.096683
> mon/OSDMonitor.cc: 156: FAILED assert(latest_full > 0)
>
> ceph version 0.61.6-15-g85db066 (85db0667307ac803c753d16fa374dd2fc29d76f3)
> 1: (OSDMonitor::update_from_paxos(bool*)+0x2413) [0x50f5a3]
> 2: (PaxosService::refresh(bool*)+0xe6) [0x4f2c66]
> 3: (Monitor::refresh_from_paxos(bool*)+0x57) [0x48f7b7]
> 4: (Monitor::init_paxos()+0xe5) [0x48f955]
> 5: (Monitor::preinit()+0x679) [0x4bba79]
> 6: (main()+0x36b0) [0x484bb0]
> 7: (__libc_start_main()+0xfd) [0x7fae619a6c8d]
> 8: /usr/bin/ceph-mon() [0x4801e9]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
>
> --- begin dump of recent events ---
> -13> 2013-07-24 08:41:44.222821 7fae6384a780 5 asok(0x2698000)
> register_command perfcounters_dump hook 0x2682010
> -12> 2013-07-24 08:41:44.222835 7fae6384a780 5 asok(0x2698000)
> register_command 1 hook 0x2682010
> -11> 2013-07-24 08:41:44.222837 7fae6384a780 5 asok(0x2698000)
> register_command perf dump hook 0x2682010
> -10> 2013-07-24 08:41:44.222842 7fae6384a780 5 asok(0x2698000)
> register_command perfcounters_schema hook 0x2682010
> -9> 2013-07-24 08:41:44.222845 7fae6384a780 5 asok(0x2698000)
> register_command 2 hook 0x2682010
> -8> 2013-07-24 08:41:44.222847 7fae6384a780 5 asok(0x2698000)
> register_command perf schema hook 0x2682010
> -7> 2013-07-24 08:41:44.222849 7fae6384a780 5 asok(0x2698000)
> register_command config show hook 0x2682010
> -6> 2013-07-24 08:41:44.222852 7fae6384a780 5 asok(0x2698000)
> register_command config set hook 0x2682010
> -5> 2013-07-24 08:41:44.222854 7fae6384a780 5 asok(0x2698000)
> register_command log flush hook 0x2682010
> -4> 2013-07-24 08:41:44.222856 7fae6384a780 5 asok(0x2698000)
> register_command log dump hook 0x2682010
> -3> 2013-07-24 08:41:44.222859 7fae6384a780 5 asok(0x2698000)
> register_command log reopen hook 0x2682010
> -2> 2013-07-24 08:41:44.224104 7fae6384a780 0 ceph version
> 0.61.6-15-g85db066 (85db0667307ac803c753d16fa374dd2fc29d76f3), process
> ceph-mon, pid 29871
> -1> 2013-07-24 08:41:44.224397 7fae6384a780 1 finished
> global_init_daemonize
> 0> 2013-07-24 08:41:56.097385 7fae6384a780 -1 mon/OSDMonitor.cc: In
> function 'virtual void OSDMonitor::update_from_paxos(bool*)' thread
> 7fae6384a780 time 2013-07-24 08:41:56.096683
> mon/OSDMonitor.cc: 156: FAILED assert(latest_full > 0)
>
> ceph version 0.61.6-15-g85db066 (85db0667307ac803c753d16fa374dd2fc29d76f3)
> 1: (OSDMonitor::update_from_paxos(bool*)+0x2413) [0x50f5a3]
> 2: (PaxosService::refresh(bool*)+0xe6) [0x4f2c66]
> 3: (Monitor::refresh_from_paxos(bool*)+0x57) [0x48f7b7]
> 4: (Monitor::init_paxos()+0xe5) [0x48f955]
> 5: (Monitor::preinit()+0x679) [0x4bba79]
> 6: (main()+0x36b0) [0x484bb0]
> 7: (__libc_start_main()+0xfd) [0x7fae619a6c8d]
> 8: /usr/bin/ceph-mon() [0x4801e9]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
>
> 4.) i thought no problem mon.b and mon.c are still running. BUT all OSDs
> were still trying to reach mon.a
>
> 2013-07-24 08:41:43.088997 7f011268f700 0 monclient: hunting for new mon
> 2013-07-24 08:41:56.792449 7f0109e7e700 0 -- 10.255.0.82:6802/29397 >>
> 10.255.0.100:6789/0 pipe(0x489e000 sd=286 :0 s=1 pgs=0 cs=0 l=1).fault
> 2013-07-24 08:42:02.792990 7f0116b6c700 0 -- 10.255.0.82:6802/29397 >>
> 10.255.0.100:6789/0 pipe(0x3c02780 sd=256 :0 s=1 pgs=0 cs=0 l=1).fault
> 2013-07-24 08:42:11.793525 7f0109d7d700 0 -- 10.255.0.82:6802/29397 >>
> 10.255.0.100:6789/0 pipe(0x84ec280 sd=256 :0 s=1 pgs=0 cs=0 l=1).fault
> 2013-07-24 08:42:23.794315 7f0109e7e700 0 -- 10.255.0.82:6802/29397 >>
> 10.255.0.100:6789/0 pipe(0x44c7b80 sd=286 :0 s=1 pgs=0 cs=0 l=1).fault
> 2013-07-24 08:42:27.621336 7f0122d2e700 0 log [WRN] : 5 slow requests,
> 5 included below; oldest blocked for > 30.378391 secs
> 2013-07-24 08:42:27.621344 7f0122d2e700 0 log [WRN] : slow request
> 30.378391 seconds old, received at 2013-07-24 08:41:57.242902:
> osd_op(client.14727601.0:3839848
> rbd_data.e0b5b26b8b4567.0000000000005b5a [write 684032~4096] 5.816d89d1
> snapc bef=[bef] e142137) v4 currently wait for new map
> 2013-07-24 08:42:27.621348 7f0122d2e700 0 log [WRN] : slow request
> 30.195074 seconds old, received at 2013-07-24 08:41:57.426219:
> osd_op(client.14828945.0:1088870
> rbd_data.e245696b8b4567.000000000000140e [write 988160~7168] 5.ed959c36
> snapc b80=[b80] e142137) v4 currently wait for new map
> 2013-07-24 08:42:27.621350 7f0122d2e700 0 log [WRN] : slow request
> 30.148871 seconds old, received at 2013-07-24 08:41:57.472422:
> osd_op(client.14667314.0:2818172
> rbd_data.dfcaa86b8b4567.0000000000000a13 [write 1654784~4096] 5.6972a67e
> snapc baa=[baa] e142137) v4 currently wait for new map
> 2013-07-24 08:42:27.621351 7f0122d2e700 0 log [WRN] : slow request
> 30.148829 seconds old, received at 2013-07-24 08:41:57.472464:
> osd_op(client.14667314.0:2818173
> rbd_data.dfcaa86b8b4567.0000000000000a13 [write 1957888~4096] 5.6972a67e
> snapc baa=[baa] e142137) v4 currently wait for new map
> 2013-07-24 08:42:27.621352 7f0122d2e700 0 log [WRN] : slow request
> 30.148784 seconds old, received at 2013-07-24 08:41:57.472509:
> osd_op(client.14667314.0:2818174
> rbd_data.dfcaa86b8b4567.0000000000000a13 [write 1966080~4096] 5.6972a67e
> snapc baa=[baa] e142137) v4 currently wait for new map
>
> ...
>
> 2013-07-24 08:50:20.826687 7f00ee6d9700 0 -- 10.255.0.82:6802/29397 >>
> 10.255.0.100:6789/0 pipe(0xdf02280 sd=288 :0 s=1 pgs=0 cs=0 l=1).fault
> 2013-07-24 08:50:26.826914 7f00f1697700 0 -- 10.255.0.82:6802/29397 >>
> 10.255.0.100:6789/0 pipe(0x465a000 sd=229 :0 s=1 pgs=0 cs=0 l=1).fault
> 2013-07-24 08:50:40.713100 7f00ee6d9700 0 -- 10.255.0.82:6802/29397 >>
> 10.255.0.100:6789/0 pipe(0x4383680 sd=281 :0 s=1 pgs=0 cs=0 l=1).fault
> 2013-07-24 08:50:44.828164 7f011392a700 0 -- 10.255.0.82:6802/29397 >>
> 10.255.0.100:6789/0 pipe(0x41ecf00 sd=281 :0 s=1 pgs=0 cs=0 l=1).fault
> 2013-07-24 08:51:02.829357 7f00f1697700 0 -- 10.255.0.82:6802/29397 >>
> 10.255.0.100:6789/0 pipe(0x1d8b180 sd=281 :0 s=1 pgs=0 cs=0 l=1).fault
>
> Stefan
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
