Re: Monitor is unable to start after reboot: OSDMonitor::update_from_paxos(bool*) FAILED assert(latest_bl.length() != 0

[peter@xxxxxxxxx]

Hi Joao,

I have sent you the link to the monitor files. I stopped one other 
monitor to have a consistent tarball but now it won't start, crashing 
with the same error message. I hope there is a trick to get it working 
again because now I only have one monitor working and I don't want to 
end up losing data again (I had this happen once before).

Thanks,

Peter

On 2013-07-22 17:31, Joao Eduardo Luis wrote:
> On 07/22/2013 12:33 PM, peter@xxxxxxxxx wrote:
> > Hello,
> >
> > After a reboot one of our monitors is unable to start. We did an 
> > upgrade
> > from 0.61.4 to 0.61.5 last week without problems (the monitor 
> > restarted
> > just fine).
> >
> > We are getting the following error (I think it is the same as:
> > http://tracker.ceph.com/issues/5704). I might have missed it on the 
> > list
> > though. If you want I can send the contents of the monitor directory.
>
> That monitor store would be greatly appreciated!  If you could bundle
> the store of two other monitors it would be great.
>
>   -Joao
>
>
> > 2013-07-22 13:24:02.183558 7fd06127e780  0 ceph version 0.61.5
> > (8ee10dc4bb73bdd918873f29c70eedc3c7ef1979), process ceph-mon, pid 
> > 28540
> > 2013-07-22 13:24:02.251205 7fd05d320700 -1 asok(0x207e000) 
> > AdminSocket:
> > request 'mon_status' not defined
> > 2013-07-22 13:24:02.357287 7fd06127e780  1 mon.narr9@-1(probing) e1
> > preinit fsid 97e515bb-d334-4fa7-8b53-7d85615809fd
> > 2013-07-22 13:24:02.374158 7fd06127e780 -1 mon/OSDMonitor.cc: In
> > function 'virtual void OSDMonitor::update_from_paxos(bool*)' thread
> > 7fd06127e780 time 2013-07-22 13:24:02.373344
> > mon/OSDMonitor.cc: 132: FAILED assert(latest_bl.length() != 0)
> >
> >   ceph version 0.61.5 (8ee10dc4bb73bdd918873f29c70eedc3c7ef1979)
> >   1: /usr/bin/ceph-mon() [0x5073d6]
> >   2: (PaxosService::refresh(bool*)+0x19b) [0x4edd4b]
> >   3: (Monitor::refresh_from_paxos(bool*)+0x57) [0x48e5a7]
> >   4: (Monitor::init_paxos()+0xf5) [0x48e755]
> >   5: (Monitor::preinit()+0x6ac) [0x4a4e7c]
> >   6: (main()+0x1c19) [0x483559]
> >   7: (__libc_start_main()+0xed) [0x7fd05f4da76d]
> >   8: /usr/bin/ceph-mon() [0x485e7d]
> >   NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> > needed to interpret this.
> >
> > --- begin dump of recent events ---
> >     -26> 2013-07-22 13:24:02.181870 7fd06127e780  5 asok(0x207e000)
> > register_command perfcounters_dump hook 0x2073010
> >     -25> 2013-07-22 13:24:02.181908 7fd06127e780  5 asok(0x207e000)
> > register_command 1 hook 0x2073010
> >     -24> 2013-07-22 13:24:02.181915 7fd06127e780  5 asok(0x207e000)
> > register_command perf dump hook 0x2073010
> >     -23> 2013-07-22 13:24:02.181929 7fd06127e780  5 asok(0x207e000)
> > register_command perfcounters_schema hook 0x2073010
> >     -22> 2013-07-22 13:24:02.181939 7fd06127e780  5 asok(0x207e000)
> > register_command 2 hook 0x2073010
> >     -21> 2013-07-22 13:24:02.181941 7fd06127e780  5 asok(0x207e000)
> > register_command perf schema hook 0x2073010
> >     -20> 2013-07-22 13:24:02.181945 7fd06127e780  5 asok(0x207e000)
> > register_command config show hook 0x2073010
> >     -19> 2013-07-22 13:24:02.181954 7fd06127e780  5 asok(0x207e000)
> > register_command config set hook 0x2073010
> >     -18> 2013-07-22 13:24:02.181957 7fd06127e780  5 asok(0x207e000)
> > register_command log flush hook 0x2073010
> >     -17> 2013-07-22 13:24:02.181959 7fd06127e780  5 asok(0x207e000)
> > register_command log dump hook 0x2073010
> >     -16> 2013-07-22 13:24:02.181964 7fd06127e780  5 asok(0x207e000)
> > register_command log reopen hook 0x2073010
> >     -15> 2013-07-22 13:24:02.183558 7fd06127e780  0 ceph version 
> > 0.61.5
> > (8ee10dc4bb73bdd918873f29c70eedc3c7ef1979), process ceph-mon, pid 
> > 28540
> >     -14> 2013-07-22 13:24:02.186703 7fd06127e780  5 asok(0x207e000) 
> > init
> > /var/run/ceph/ceph-mon.narr9.asok
> >     -13> 2013-07-22 13:24:02.186734 7fd06127e780  5 asok(0x207e000)
> > bind_and_listen /var/run/ceph/ceph-mon.narr9.asok
> >     -12> 2013-07-22 13:24:02.186780 7fd06127e780  5 asok(0x207e000)
> > register_command 0 hook 0x20720b0
> >     -11> 2013-07-22 13:24:02.186790 7fd06127e780  5 asok(0x207e000)
> > register_command version hook 0x20720b0
> >     -10> 2013-07-22 13:24:02.186798 7fd06127e780  5 asok(0x207e000)
> > register_command git_version hook 0x20720b0
> >      -9> 2013-07-22 13:24:02.186806 7fd06127e780  5 asok(0x207e000)
> > register_command help hook 0x20730d0
> >      -8> 2013-07-22 13:24:02.186850 7fd05d320700  5 asok(0x207e000)
> > entry start
> >      -7> 2013-07-22 13:24:02.251205 7fd05d320700 -1 asok(0x207e000)
> > AdminSocket: request 'mon_status' not defined
> >      -6> 2013-07-22 13:24:02.357202 7fd06127e780  1 --
> > 10.255.0.25:6789/0 learned my addr 10.255.0.25:6789/0
> >      -5> 2013-07-22 13:24:02.357215 7fd06127e780  1
> > accepter.accepter.bind my_inst.addr is 10.255.0.25:6789/0 need_addr=0
> >      -4> 2013-07-22 13:24:02.357242 7fd06127e780  5 adding auth
> > protocol: cephx
> >      -3> 2013-07-22 13:24:02.357245 7fd06127e780  5 adding auth
> > protocol: cephx
> >      -2> 2013-07-22 13:24:02.357287 7fd06127e780  1
> > mon.narr9@-1(probing) e1 preinit fsid 
> > 97e515bb-d334-4fa7-8b53-7d85615809fd
> >      -1> 2013-07-22 13:24:02.372987 7fd06127e780  4
> > mon.narr9@-1(probing).mds e182116 new map
> >       0> 2013-07-22 13:24:02.374158 7fd06127e780 -1 
> > mon/OSDMonitor.cc:
> > In function 'virtual void OSDMonitor::update_from_paxos(bool*)' 
> > thread
> > 7fd06127e780 time 2013-07-22 13:24:02.373344
> > mon/OSDMonitor.cc: 132: FAILED assert(latest_bl.length() != 0)
> >
> >   ceph version 0.61.5 (8ee10dc4bb73bdd918873f29c70eedc3c7ef1979)
> >   1: /usr/bin/ceph-mon() [0x5073d6]
> >   2: (PaxosService::refresh(bool*)+0x19b) [0x4edd4b]
> >   3: (Monitor::refresh_from_paxos(bool*)+0x57) [0x48e5a7]
> >   4: (Monitor::init_paxos()+0xf5) [0x48e755]
> >   5: (Monitor::preinit()+0x6ac) [0x4a4e7c]
> >   6: (main()+0x1c19) [0x483559]
> >   7: (__libc_start_main()+0xed) [0x7fd05f4da76d]
> >   8: /usr/bin/ceph-mon() [0x485e7d]
> >   NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> > needed to interpret this.
> >
> > --- logging levels ---
> >     0/ 5 none
> >     0/ 1 lockdep
> >     0/ 1 context
> >     1/ 1 crush
> >     1/ 5 mds
> >     1/ 5 mds_balancer
> >     1/ 5 mds_locker
> >     1/ 5 mds_log
> >     1/ 5 mds_log_expire
> >     1/ 5 mds_migrator
> >     0/ 1 buffer
> >     0/ 1 timer
> >     0/ 1 filer
> >     0/ 1 striper
> >     0/ 1 objecter
> >     0/ 5 rados
> >     0/ 5 rbd
> >     0/ 5 journaler
> >     0/ 5 objectcacher
> >     0/ 5 client
> >     0/ 5 osd
> >     0/ 5 optracker
> >     0/ 5 objclass
> >     1/ 3 filestore
> >     1/ 3 journal
> >     0/ 5 ms
> >     1/ 5 mon
> >     0/10 monc
> >     0/ 5 paxos
> >     0/ 5 tp
> >     1/ 5 auth
> >     1/ 5 crypto
> >     1/ 1 finisher
> >     1/ 5 heartbeatmap
> >     1/ 5 perfcounter
> >     1/ 5 rgw
> >     1/ 5 hadoop
> >     1/ 5 javaclient
> >     1/ 5 asok
> >     1/ 1 throttle
> >    -2/-2 (syslog threshold)
> >    -1/-1 (stderr threshold)
> >    max_recent     10000
> >    max_new         1000
> >    log_file /var/log/ceph/ceph-mon.narr9.log
> > --- end dump of recent events ---
> > 2013-07-22 13:24:02.376004 7fd06127e780 -1 *** Caught signal 
> > (Aborted) **
> >   in thread 7fd06127e780
> >
> >   ceph version 0.61.5 (8ee10dc4bb73bdd918873f29c70eedc3c7ef1979)
> >   1: /usr/bin/ceph-mon() [0x59743a]
> >   2: (()+0xfcb0) [0x7fd060919cb0]
> >   3: (gsignal()+0x35) [0x7fd05f4ef425]
> >   4: (abort()+0x17b) [0x7fd05f4f2b8b]
> >   5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) 
> > [0x7fd05fe4169d]
> >   6: (()+0xb5846) [0x7fd05fe3f846]
> >   7: (()+0xb5873) [0x7fd05fe3f873]
> >   8: (()+0xb596e) [0x7fd05fe3f96e]
> >   9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> > const*)+0x1df) [0x64f6ef]
> >   10: /usr/bin/ceph-mon() [0x5073d6]
> >   11: (PaxosService::refresh(bool*)+0x19b) [0x4edd4b]
> >   12: (Monitor::refresh_from_paxos(bool*)+0x57) [0x48e5a7]
> >   13: (Monitor::init_paxos()+0xf5) [0x48e755]
> >   14: (Monitor::preinit()+0x6ac) [0x4a4e7c]
> >   15: (main()+0x1c19) [0x483559]
> >   16: (__libc_start_main()+0xed) [0x7fd05f4da76d]
> >   17: /usr/bin/ceph-mon() [0x485e7d]
> >   NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> > needed to interpret this.
> >
> > --- begin dump of recent events ---
> >       0> 2013-07-22 13:24:02.376004 7fd06127e780 -1 *** Caught signal
> > (Aborted) **
> >   in thread 7fd06127e780
> >
> >   ceph version 0.61.5 (8ee10dc4bb73bdd918873f29c70eedc3c7ef1979)
> >   1: /usr/bin/ceph-mon() [0x59743a]
> >   2: (()+0xfcb0) [0x7fd060919cb0]
> >   3: (gsignal()+0x35) [0x7fd05f4ef425]
> >   4: (abort()+0x17b) [0x7fd05f4f2b8b]
> >   5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) 
> > [0x7fd05fe4169d]
> >   6: (()+0xb5846) [0x7fd05fe3f846]
> >   7: (()+0xb5873) [0x7fd05fe3f873]
> >   8: (()+0xb596e) [0x7fd05fe3f96e]
> >   9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> > const*)+0x1df) [0x64f6ef]
> >   10: /usr/bin/ceph-mon() [0x5073d6]
> >   11: (PaxosService::refresh(bool*)+0x19b) [0x4edd4b]
> >   12: (Monitor::refresh_from_paxos(bool*)+0x57) [0x48e5a7]
> >   13: (Monitor::init_paxos()+0xf5) [0x48e755]
> >   14: (Monitor::preinit()+0x6ac) [0x4a4e7c]
> >   15: (main()+0x1c19) [0x483559]
> >   16: (__libc_start_main()+0xed) [0x7fd05f4da76d]
> >   17: /usr/bin/ceph-mon() [0x485e7d]
> >   NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> > needed to interpret this.
> >
> > --- logging levels ---
> >     0/ 5 none
> >     0/ 1 lockdep
> >     0/ 1 context
> >     1/ 1 crush
> >     1/ 5 mds
> >     1/ 5 mds_balancer
> >     1/ 5 mds_locker
> >     1/ 5 mds_log
> >     1/ 5 mds_log_expire
> >     1/ 5 mds_migrator
> >     0/ 1 buffer
> >     0/ 1 timer
> >     0/ 1 filer
> >     0/ 1 striper
> >     0/ 1 objecter
> >     0/ 5 rados
> >     0/ 5 rbd
> >     0/ 5 journaler
> >     0/ 5 objectcacher
> >     0/ 5 client
> >     0/ 5 osd
> >     0/ 5 optracker
> >     0/ 5 objclass
> >     1/ 3 filestore
> >     1/ 3 journal
> >     0/ 5 ms
> >     1/ 5 mon
> >     0/10 monc
> >     0/ 5 paxos
> >     0/ 5 tp
> >     1/ 5 auth
> >     1/ 5 crypto
> >     1/ 1 finisher
> >     1/ 5 heartbeatmap
> >     1/ 5 perfcounter
> >     1/ 5 rgw
> >     1/ 5 hadoop
> >     1/ 5 javaclient
> >     1/ 5 asok
> >     1/ 1 throttle
> >    -2/-2 (syslog threshold)
> >    -1/-1 (stderr threshold)
> >    max_recent     10000
> >    max_new         1000
> >    log_file /var/log/ceph/ceph-mon.narr9.log
> > --- end dump of recent events ---
> >
> > Cheers,
> >
> > Peter
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@xxxxxxxxxxxxxx
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com