Re: OSDs flapping

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

At a glance that looks like the bug fixed by just-merged https://github.com/ceph/ceph/pull/16421

On Thu, Jul 20, 2017 at 1:02 PM Roger Brown <rogerpbrown@xxxxxxxxx> wrote:
I'm on Luminous 12.1.1 and noticed I have flapping OSDs. Even with `ceph osd set nodown`, the OSDs will catch signal Aborted and sometimes Segmentation fault 2-5 minutes after starting. I verified hosts can talk to eachother on the cluster network. I've rebooted the hosts. I'm running out of ideas. Please advise.

Tally of crashes:
roger@osd1:~$ sudo grep ': \*\*\* Caught signal' /var/log/syslog{,.1} | awk '{print $9}' | sort | uniq -c | sort -nr
    100 (Segmentation
     77 (Aborted)
roger@osd2:~$ sudo grep ': \*\*\* Caught signal' /var/log/syslog{,.1} | awk '{print $9}' | sort | uniq -c | sort -nr
     77 (Aborted)
     13 (Segmentation
roger@osd3:~$ sudo grep ': \*\*\* Caught signal' /var/log/syslog{,.1} | awk '{print $9}' | sort | uniq -c | sort -nr
     86 (Aborted)
      3 (Segmentation

First crash observed Jul 19:
roger@osd1:~$ sudo grep ': \*\*\* Caught signal' /var/log/syslog.1 | head -1
Jul 19 10:07:12 osd1 ceph-osd[13491]: *** Caught signal (Aborted) **
roger@osd2:~$ sudo grep ': \*\*\* Caught signal' /var/log/syslog.1 | head -1
Jul 19 10:07:36 osd2 ceph-osd[13937]: *** Caught signal (Aborted) **
roger@osd3:~$ sudo grep ': \*\*\* Caught signal' /var/log/syslog.1 | head -1
Jul 19 16:07:12 osd3 ceph-osd[8807]: *** Caught signal (Aborted) **

Crashes started with Luminous 12.1.0:
roger@osd1:~$ sudo grep 'Jul 19 10:07:12.*ceph version' /var/log/syslog.1 | head -1
Jul 19 10:07:12 osd1 ceph-osd[13491]:  ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)
roger@osd2:~$ sudo grep 'Jul 19 10:07:36.*ceph version' /var/log/syslog.1 | head -1
Jul 19 10:07:36 osd2 ceph-osd[13937]:  ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)
roger@osd3:~$ sudo grep 'Jul 19 16:07:12.*ceph version' /var/log/syslog.1 | head -1
Jul 19 16:07:12 osd3 ceph-osd[8807]:  ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)

Representative example from osd1 logs:
Jul 20 13:42:18 osd1 ceph-osd[4035]: *** Caught signal (Segmentation fault) **
Jul 20 13:42:18 osd1 ceph-osd[4035]:  in thread 7f52960e7700 thread_name:msgr-worker-2
Jul 20 13:42:18 osd1 ceph-osd[4035]: 2017-07-20 13:42:18.658076 7f529bf85c80 -1 osd.3 3444 log_to_monitors {default=true}
Jul 20 13:42:18 osd1 ceph-osd[4035]: 2017-07-20 13:42:18.662695 7f52968e8700 -1 failed to decode message of type 70 v3: buffer::malformed_input: void osd_peer_stat_t::decode(ceph::buffer::list::iterator&) no longer understand old encoding version 1 < struct_compat
Jul 20 13:42:18 osd1 ceph-osd[4035]:  ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
Jul 20 13:42:18 osd1 ceph-osd[4035]:  1: (()+0xa257a4) [0x55bc98fe27a4]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  2: (()+0x11390) [0x7f529a468390]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  3: (cephx_verify_authorizer(CephContext*, KeyStore*, ceph::buffer::list::iterator&, CephXServiceTicketInfo&, ceph::buffer::list&)+0x496) [0x55bc991b0ca6]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  4: (CephxAuthorizeHandler::verify_authorizer(CephContext*, KeyStore*, ceph::buffer::list&, ceph::buffer::list&, EntityName&, unsigned long&, AuthCapsInfo&, CryptoKey&, unsigned long*)+0x31a) [0x55bc991a2cda]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  5: (OSD::ms_verify_authorizer(Connection*, int, int, ceph::buffer::list&, ceph::buffer::list&, bool&, CryptoKey&)+0xf9) [0x55bc98a2c759]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  6: (AsyncConnection::handle_connect_msg(ceph_msg_connect&, ceph::buffer::list&, ceph::buffer::list&)+0x228) [0x55bc99271108]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  7: (AsyncConnection::_process_connection()+0x1e07) [0x55bc99276a57]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  8: (AsyncConnection::process()+0x1ae8) [0x55bc9927b978]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  9: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x55bc990c6148]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  10: (()+0xb0d0d8) [0x55bc990ca0d8]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  11: (()+0xb8c80) [0x7f5299d6fc80]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  12: (()+0x76ba) [0x7f529a45e6ba]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  13: (clone()+0x6d) [0x7f52994d53dd]
Jul 20 13:42:18 osd1 ceph-osd[4035]: 2017-07-20 13:42:18.662763 7f52960e7700 -1 *** Caught signal (Segmentation fault) **
Jul 20 13:42:18 osd1 ceph-osd[4035]:  in thread 7f52960e7700 thread_name:msgr-worker-2
Jul 20 13:42:18 osd1 ceph-osd[4035]:  ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
Jul 20 13:42:18 osd1 ceph-osd[4035]:  1: (()+0xa257a4) [0x55bc98fe27a4]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  2: (()+0x11390) [0x7f529a468390]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  3: (cephx_verify_authorizer(CephContext*, KeyStore*, ceph::buffer::list::iterator&, CephXServiceTicketInfo&, ceph::buffer::list&)+0x496) [0x55bc991b0ca6]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  4: (CephxAuthorizeHandler::verify_authorizer(CephContext*, KeyStore*, ceph::buffer::list&, ceph::buffer::list&, EntityName&, unsigned long&, AuthCapsInfo&, CryptoKey&, unsigned long*)+0x31a) [0x55bc991a2cda]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  5: (OSD::ms_verify_authorizer(Connection*, int, int, ceph::buffer::list&, ceph::buffer::list&, bool&, CryptoKey&)+0xf9) [0x55bc98a2c759]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  6: (AsyncConnection::handle_connect_msg(ceph_msg_connect&, ceph::buffer::list&, ceph::buffer::list&)+0x228) [0x55bc99271108]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  7: (AsyncConnection::_process_connection()+0x1e07) [0x55bc99276a57]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  8: (AsyncConnection::process()+0x1ae8) [0x55bc9927b978]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  9: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x55bc990c6148]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  10: (()+0xb0d0d8) [0x55bc990ca0d8]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  11: (()+0xb8c80) [0x7f5299d6fc80]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  12: (()+0x76ba) [0x7f529a45e6ba]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  13: (clone()+0x6d) [0x7f52994d53dd]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jul 20 13:42:18 osd1 ceph-osd[4035]:    -18> 2017-07-20 13:42:18.658076 7f529bf85c80 -1 osd.3 3444 log_to_monitors {default=true}
Jul 20 13:42:18 osd1 ceph-osd[4035]:     -5> 2017-07-20 13:42:18.662695 7f52968e8700 -1 failed to decode message of type 70 v3: buffer::malformed_input: void osd_peer_stat_t::decode(ceph::buffer::list::iterator&) no longer understand old encoding version 1 < struct_compat
Jul 20 13:42:18 osd1 ceph-osd[4035]:      0> 2017-07-20 13:42:18.662763 7f52960e7700 -1 *** Caught signal (Segmentation fault) **
Jul 20 13:42:18 osd1 ceph-osd[4035]:  in thread 7f52960e7700 thread_name:msgr-worker-2
Jul 20 13:42:18 osd1 ceph-osd[4035]:  ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
Jul 20 13:42:18 osd1 ceph-osd[4035]:  1: (()+0xa257a4) [0x55bc98fe27a4]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  2: (()+0x11390) [0x7f529a468390]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  3: (cephx_verify_authorizer(CephContext*, KeyStore*, ceph::buffer::list::iterator&, CephXServiceTicketInfo&, ceph::buffer::list&)+0x496) [0x55bc991b0ca6]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  4: (CephxAuthorizeHandler::verify_authorizer(CephContext*, KeyStore*, ceph::buffer::list&, ceph::buffer::list&, EntityName&, unsigned long&, AuthCapsInfo&, CryptoKey&, unsigned long*)+0x31a) [0x55bc991a2cda]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  5: (OSD::ms_verify_authorizer(Connection*, int, int, ceph::buffer::list&, ceph::buffer::list&, bool&, CryptoKey&)+0xf9) [0x55bc98a2c759]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  6: (AsyncConnection::handle_connect_msg(ceph_msg_connect&, ceph::buffer::list&, ceph::buffer::list&)+0x228) [0x55bc99271108]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  7: (AsyncConnection::_process_connection()+0x1e07) [0x55bc99276a57]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  8: (AsyncConnection::process()+0x1ae8) [0x55bc9927b978]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  9: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x55bc990c6148]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  10: (()+0xb0d0d8) [0x55bc990ca0d8]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  11: (()+0xb8c80) [0x7f5299d6fc80]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  12: (()+0x76ba) [0x7f529a45e6ba]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  13: (clone()+0x6d) [0x7f52994d53dd]
Jul 20 13:42:18 osd1 ceph-osd[4035]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jul 20 13:42:18 osd1 systemd[1]: ceph-osd@3.service: Main process exited, code=killed, status=11/SEGV
Jul 20 13:42:18 osd1 systemd[1]: ceph-osd@3.service: Unit entered failed state.
Jul 20 13:42:18 osd1 systemd[1]: ceph-osd@3.service: Failed with result 'signal'.
Jul 20 13:42:38 osd1 systemd[1]: ceph-osd@3.service: Service hold-off time over, scheduling restart.
Jul 20 13:42:38 osd1 systemd[1]: Stopped Ceph object storage daemon osd.3.
Jul 20 13:42:38 osd1 systemd[1]: Starting Ceph object storage daemon osd.3...
Jul 20 13:42:39 osd1 systemd[1]: Started Ceph object storage daemon osd.3.
Jul 20 13:42:39 osd1 ceph-osd[4130]: starting osd.3 at - osd_data /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
Jul 20 13:43:02 osd1 sshd[3497]: Received disconnect from 192.168.0.7 port 55258:11: disconnected by user
Jul 20 13:43:02 osd1 sshd[3497]: Disconnected from 192.168.0.7 port 55258
Jul 20 13:43:02 osd1 sshd[3466]: pam_unix(sshd:session): session closed for user roger
Jul 20 13:43:02 osd1 systemd-logind[1393]: Removed session 10.
Jul 20 13:44:53 osd1 ceph-osd[4130]: 2017-07-20 13:44:53.540934 7f303995dc80 -1 osd.3 3444 log_to_monitors {default=true}
Jul 20 13:45:33 osd1 ceph-osd[4130]: 2017-07-20 13:45:33.544688 7f30302de700 -1 osd.3 3458 heartbeat_check: no reply from 192.168.0.26:6801 osd.0 since back 2017-07-20 13:45:12.643355 front 2017-07-20 13:45:12.643355 (cutoff 2017-07-20 13:45:13.544686)
Jul 20 13:46:01 osd1 ceph-osd[4130]: /build/ceph-12.1.1/src/osd/osd_types.h: In function 'static void osd_reqid_t::_denc_finish(ceph::buffer::ptr::iterator&, __u8*, __u8*, char**, uint32_t*)' thread 7f3033abf700 time 2017-07-20 13:46:01.429584
Jul 20 13:46:01 osd1 ceph-osd[4130]: /build/ceph-12.1.1/src/osd/osd_types.h: 117: FAILED assert(pos <= end)
Jul 20 13:46:01 osd1 ceph-osd[4130]:  ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
Jul 20 13:46:01 osd1 ceph-osd[4130]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55f2073ffb72]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  2: (std::enable_if<denc_traits<osd_reqid_t, void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous, void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x55f206f125d9]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  3: (MOSDRepOp::decode_payload()+0x9e) [0x55f20711a1ce]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  4: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8) [0x55f20743a5d8]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  5: (AsyncConnection::process()+0x1d4e) [0x55f207656bde]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  6: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x55f2074a1148]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  7: (()+0xb0d0d8) [0x55f2074a50d8]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  8: (()+0xb8c80) [0x7f3037747c80]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  9: (()+0x76ba) [0x7f3037e366ba]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  10: (clone()+0x6d) [0x7f3036ead3dd]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jul 20 13:46:01 osd1 ceph-osd[4130]: 2017-07-20 13:46:01.434169 7f3033abf700 -1 /build/ceph-12.1.1/src/osd/osd_types.h: In function 'static void osd_reqid_t::_denc_finish(ceph::buffer::ptr::iterator&, __u8*, __u8*, char**, uint32_t*)' thread 7f3033abf700 time 2017-07-20 13:46:01.429584
Jul 20 13:46:01 osd1 ceph-osd[4130]: /build/ceph-12.1.1/src/osd/osd_types.h: 117: FAILED assert(pos <= end)
Jul 20 13:46:01 osd1 ceph-osd[4130]:  ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
Jul 20 13:46:01 osd1 ceph-osd[4130]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55f2073ffb72]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  2: (std::enable_if<denc_traits<osd_reqid_t, void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous, void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x55f206f125d9]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  3: (MOSDRepOp::decode_payload()+0x9e) [0x55f20711a1ce]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  4: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8) [0x55f20743a5d8]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  5: (AsyncConnection::process()+0x1d4e) [0x55f207656bde]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  6: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x55f2074a1148]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  7: (()+0xb0d0d8) [0x55f2074a50d8]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  8: (()+0xb8c80) [0x7f3037747c80]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  9: (()+0x76ba) [0x7f3037e366ba]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  10: (clone()+0x6d) [0x7f3036ead3dd]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jul 20 13:46:01 osd1 ceph-osd[4130]:      0> 2017-07-20 13:46:01.434169 7f3033abf700 -1 /build/ceph-12.1.1/src/osd/osd_types.h: In function 'static void osd_reqid_t::_denc_finish(ceph::buffer::ptr::iterator&, __u8*, __u8*, char**, uint32_t*)' thread 7f3033abf700 time 2017-07-20 13:46:01.429584
Jul 20 13:46:01 osd1 ceph-osd[4130]: /build/ceph-12.1.1/src/osd/osd_types.h: 117: FAILED assert(pos <= end)
Jul 20 13:46:01 osd1 ceph-osd[4130]:  ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
Jul 20 13:46:01 osd1 ceph-osd[4130]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55f2073ffb72]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  2: (std::enable_if<denc_traits<osd_reqid_t, void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous, void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x55f206f125d9]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  3: (MOSDRepOp::decode_payload()+0x9e) [0x55f20711a1ce]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  4: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8) [0x55f20743a5d8]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  5: (AsyncConnection::process()+0x1d4e) [0x55f207656bde]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  6: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x55f2074a1148]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  7: (()+0xb0d0d8) [0x55f2074a50d8]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  8: (()+0xb8c80) [0x7f3037747c80]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  9: (()+0x76ba) [0x7f3037e366ba]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  10: (clone()+0x6d) [0x7f3036ead3dd]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jul 20 13:46:01 osd1 ceph-osd[4130]: *** Caught signal (Aborted) **
Jul 20 13:46:01 osd1 ceph-osd[4130]:  in thread 7f3033abf700 thread_name:msgr-worker-2
Jul 20 13:46:01 osd1 ceph-osd[4130]:  ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
Jul 20 13:46:01 osd1 ceph-osd[4130]:  1: (()+0xa257a4) [0x55f2073bd7a4]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  2: (()+0x11390) [0x7f3037e40390]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  3: (gsignal()+0x38) [0x7f3036ddb428]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  4: (abort()+0x16a) [0x7f3036ddd02a]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x55f2073ffcfe]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  6: (std::enable_if<denc_traits<osd_reqid_t, void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous, void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x55f206f125d9]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  7: (MOSDRepOp::decode_payload()+0x9e) [0x55f20711a1ce]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  8: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8) [0x55f20743a5d8]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  9: (AsyncConnection::process()+0x1d4e) [0x55f207656bde]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  10: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x55f2074a1148]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  11: (()+0xb0d0d8) [0x55f2074a50d8]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  12: (()+0xb8c80) [0x7f3037747c80]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  13: (()+0x76ba) [0x7f3037e366ba]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  14: (clone()+0x6d) [0x7f3036ead3dd]
Jul 20 13:46:01 osd1 ceph-osd[4130]: 2017-07-20 13:46:01.481468 7f3033abf700 -1 *** Caught signal (Aborted) **
Jul 20 13:46:01 osd1 ceph-osd[4130]:  in thread 7f3033abf700 thread_name:msgr-worker-2
Jul 20 13:46:01 osd1 ceph-osd[4130]:  ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
Jul 20 13:46:01 osd1 ceph-osd[4130]:  1: (()+0xa257a4) [0x55f2073bd7a4]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  2: (()+0x11390) [0x7f3037e40390]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  3: (gsignal()+0x38) [0x7f3036ddb428]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  4: (abort()+0x16a) [0x7f3036ddd02a]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x55f2073ffcfe]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  6: (std::enable_if<denc_traits<osd_reqid_t, void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous, void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x55f206f125d9]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  7: (MOSDRepOp::decode_payload()+0x9e) [0x55f20711a1ce]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  8: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8) [0x55f20743a5d8]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  9: (AsyncConnection::process()+0x1d4e) [0x55f207656bde]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  10: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x55f2074a1148]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  11: (()+0xb0d0d8) [0x55f2074a50d8]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  12: (()+0xb8c80) [0x7f3037747c80]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  13: (()+0x76ba) [0x7f3037e366ba]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  14: (clone()+0x6d) [0x7f3036ead3dd]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jul 20 13:46:01 osd1 ceph-osd[4130]:      0> 2017-07-20 13:46:01.481468 7f3033abf700 -1 *** Caught signal (Aborted) **
Jul 20 13:46:01 osd1 ceph-osd[4130]:  in thread 7f3033abf700 thread_name:msgr-worker-2
Jul 20 13:46:01 osd1 ceph-osd[4130]:  ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
Jul 20 13:46:01 osd1 ceph-osd[4130]:  1: (()+0xa257a4) [0x55f2073bd7a4]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  2: (()+0x11390) [0x7f3037e40390]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  3: (gsignal()+0x38) [0x7f3036ddb428]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  4: (abort()+0x16a) [0x7f3036ddd02a]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x55f2073ffcfe]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  6: (std::enable_if<denc_traits<osd_reqid_t, void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous, void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x55f206f125d9]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  7: (MOSDRepOp::decode_payload()+0x9e) [0x55f20711a1ce]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  8: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8) [0x55f20743a5d8]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  9: (AsyncConnection::process()+0x1d4e) [0x55f207656bde]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  10: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x55f2074a1148]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  11: (()+0xb0d0d8) [0x55f2074a50d8]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  12: (()+0xb8c80) [0x7f3037747c80]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  13: (()+0x76ba) [0x7f3037e366ba]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  14: (clone()+0x6d) [0x7f3036ead3dd]
Jul 20 13:46:01 osd1 ceph-osd[4130]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jul 20 13:46:01 osd1 systemd[1]: ceph-osd@3.service: Main process exited, code=killed, status=6/ABRT
Jul 20 13:46:01 osd1 systemd[1]: ceph-osd@3.service: Unit entered failed state.
Jul 20 13:46:01 osd1 systemd[1]: ceph-osd@3.service: Failed with result 'signal'.
Jul 20 13:46:21 osd1 systemd[1]: ceph-osd@3.service: Service hold-off time over, scheduling restart.
Jul 20 13:46:21 osd1 systemd[1]: Stopped Ceph object storage daemon osd.3.
Jul 20 13:46:21 osd1 systemd[1]: Starting Ceph object storage daemon osd.3...
Jul 20 13:46:22 osd1 systemd[1]: Started Ceph object storage daemon osd.3.
Jul 20 13:46:22 osd1 ceph-osd[4223]: starting osd.3 at - osd_data /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
Jul 20 13:48:39 osd1 ceph-osd[4223]: *** Caught signal (Segmentation fault) **
Jul 20 13:48:39 osd1 ceph-osd[4223]:  in thread 7f10b1baa700 thread_name:msgr-worker-2
Jul 20 13:48:39 osd1 ceph-osd[4223]: 2017-07-20 13:48:39.470084 7f10b7a48c80 -1 osd.3 3460 log_to_monitors {default=true}
Jul 20 13:48:39 osd1 ceph-osd[4223]:  ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
Jul 20 13:48:39 osd1 ceph-osd[4223]:  1: (()+0xa257a4) [0x55cead5be7a4]
Jul 20 13:48:39 osd1 ceph-osd[4223]:  2: (()+0x11390) [0x7f10b5f2b390]
Jul 20 13:48:39 osd1 ceph-osd[4223]:  3: (cephx_verify_authorizer(CephContext*, KeyStore*, ceph::buffer::list::iterator&, CephXServiceTicketInfo&, ceph::buffer::list&)+0x496) [0x55cead78cca6]
Jul 20 13:48:39 osd1 ceph-osd[4223]:  4: (CephxAuthorizeHandler::verify_authorizer(CephContext*, KeyStore*, ceph::buffer::list&, ceph::buffer::list&, EntityName&, unsigned long&, AuthCapsInfo&, CryptoKey&, unsigned long*)+0x31a) [0x55cead77ecda]
Jul 20 13:48:39 osd1 ceph-osd[4223]:  5: (OSD::ms_verify_authorizer(Connection*, int, int, ceph::buffer::list&, ceph::buffer::list&, bool&, CryptoKey&)+0xf9) [0x55cead008759]
Jul 20 13:48:39 osd1 ceph-osd[4223]:  6: (AsyncConnection::handle_connect_msg(ceph_msg_connect&, ceph::buffer::list&, ceph::buffer::list&)+0x228) [0x55cead84d108]
Jul 20 13:48:39 osd1 ceph-osd[4223]:  7: (AsyncConnection::_process_connection()+0x1e07) [0x55cead852a57]
Jul 20 13:48:39 osd1 ceph-osd[4223]:  7: (AsyncConnection::_process_connection()+0x1e07) [0x55cead852a57]
Jul 20 13:48:39 osd1 ceph-osd[4223]:  8: (AsyncConnection::process()+0x1ae8) [0x55cead857978]
Jul 20 13:48:39 osd1 ceph-osd[4223]:  9: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x55cead6a2148]
Jul 20 13:48:39 osd1 ceph-osd[4223]:  10: (()+0xb0d0d8) [0x55cead6a60d8]
Jul 20 13:48:39 osd1 ceph-osd[4223]:  11: (()+0xb8c80) [0x7f10b5832c80]
Jul 20 13:48:39 osd1 ceph-osd[4223]:  12: (()+0x76ba) [0x7f10b5f216ba]
Jul 20 13:48:39 osd1 ceph-osd[4223]:  13: (clone()+0x6d) [0x7f10b4f983dd]
Jul 20 13:48:39 osd1 ceph-osd[4223]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jul 20 13:48:39 osd1 systemd[1]: ceph-osd@3.service: Main process exited, code=killed, status=11/SEGV
Jul 20 13:48:39 osd1 systemd[1]: ceph-osd@3.service: Unit entered failed state.
Jul 20 13:48:39 osd1 systemd[1]: ceph-osd@3.service: Failed with result 'signal'.
Jul 20 13:48:59 osd1 systemd[1]: ceph-osd@3.service: Service hold-off time over, scheduling restart.
Jul 20 13:48:59 osd1 systemd[1]: Stopped Ceph object storage daemon osd.3.
Jul 20 13:48:59 osd1 systemd[1]: Starting Ceph object storage daemon osd.3...
Jul 20 13:49:00 osd1 systemd[1]: Started Ceph object storage daemon osd.3.
Jul 20 13:49:00 osd1 ceph-osd[4314]: starting osd.3 at - osd_data /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
Jul 20 13:51:15 osd1 ceph-osd[4314]: 2017-07-20 13:51:15.595553 7feabcf96c80 -1 osd.3 3460 log_to_monitors {default=true}
Jul 20 13:51:54 osd1 ceph-osd[4314]: 2017-07-20 13:51:54.599050 7feab3917700 -1 osd.3 3474 heartbeat_check: no reply from 192.168.0.26:6801 osd.0 since back 2017-07-20 13:51:34.297214 front 2017-07-20 13:51:34.297214 (cutoff 2017-07-20 13:51:34.599047)
Jul 20 13:51:55 osd1 ceph-osd[4314]: 2017-07-20 13:51:55.599219 7feab3917700 -1 osd.3 3474 heartbeat_check: no reply from 192.168.0.26:6801 osd.0 since back 2017-07-20 13:51:34.297214 front 2017-07-20 13:51:34.297214 (cutoff 2017-07-20 13:51:35.599217)
Jul 20 13:51:56 osd1 ceph-osd[4314]: 2017-07-20 13:51:56.599336 7feab3917700 -1 osd.3 3474 heartbeat_check: no reply from 192.168.0.26:6801 osd.0 since back 2017-07-20 13:51:34.297214 front 2017-07-20 13:51:34.297214 (cutoff 2017-07-20 13:51:36.599335)
Jul 20 13:51:57 osd1 ceph-osd[4314]: 2017-07-20 13:51:57.599445 7feab3917700 -1 osd.3 3474 heartbeat_check: no reply from 192.168.0.26:6801 osd.0 since back 2017-07-20 13:51:34.297214 front 2017-07-20 13:51:34.297214 (cutoff 2017-07-20 13:51:37.599443)
Jul 20 13:51:58 osd1 ceph-osd[4314]: 2017-07-20 13:51:58.599563 7feab3917700 -1 osd.3 3474 heartbeat_check: no reply from 192.168.0.26:6801 osd.0 since back 2017-07-20 13:51:34.297214 front 2017-07-20 13:51:34.297214 (cutoff 2017-07-20 13:51:38.599562)
Jul 20 13:52:26 osd1 ceph-osd[4314]: /build/ceph-12.1.1/src/osd/osd_types.h: In function 'static void osd_reqid_t::_denc_finish(ceph::buffer::ptr::iterator&, __u8*, __u8*, char**, uint32_t*)' thread 7feab78f9700 time 2017-07-20 13:52:26.501284
Jul 20 13:52:26 osd1 ceph-osd[4314]: /build/ceph-12.1.1/src/osd/osd_types.h: 117: FAILED assert(pos <= end)
Jul 20 13:52:26 osd1 ceph-osd[4314]:  ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
Jul 20 13:52:26 osd1 ceph-osd[4314]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x5565a2421b72]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  2: (std::enable_if<denc_traits<osd_reqid_t, void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous, void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x5565a1f345d9]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  3: (MOSDRepOp::decode_payload()+0x9e) [0x5565a213c1ce]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  4: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8) [0x5565a245c5d8]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  5: (AsyncConnection::process()+0x1d4e) [0x5565a2678bde]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  6: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x5565a24c3148]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  7: (()+0xb0d0d8) [0x5565a24c70d8]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  8: (()+0xb8c80) [0x7feabad80c80]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  9: (()+0x76ba) [0x7feabb46f6ba]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  10: (clone()+0x6d) [0x7feaba4e63dd]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jul 20 13:52:26 osd1 ceph-osd[4314]: 2017-07-20 13:52:26.505919 7feab78f9700 -1 /build/ceph-12.1.1/src/osd/osd_types.h: In function 'static void osd_reqid_t::_denc_finish(ceph::buffer::ptr::iterator&, __u8*, __u8*, char**, uint32_t*)' thread 7feab78f9700 time 2017-07-20 13:52:26.501284
Jul 20 13:52:26 osd1 ceph-osd[4314]: /build/ceph-12.1.1/src/osd/osd_types.h: 117: FAILED assert(pos <= end)
Jul 20 13:52:26 osd1 ceph-osd[4314]:  ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
Jul 20 13:52:26 osd1 ceph-osd[4314]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x5565a2421b72]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  2: (std::enable_if<denc_traits<osd_reqid_t, void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous, void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x5565a1f345d9]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  3: (MOSDRepOp::decode_payload()+0x9e) [0x5565a213c1ce]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  4: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8) [0x5565a245c5d8]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  5: (AsyncConnection::process()+0x1d4e) [0x5565a2678bde]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  6: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x5565a24c3148]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  7: (()+0xb0d0d8) [0x5565a24c70d8]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  8: (()+0xb8c80) [0x7feabad80c80]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  9: (()+0x76ba) [0x7feabb46f6ba]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  10: (clone()+0x6d) [0x7feaba4e63dd]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jul 20 13:52:26 osd1 ceph-osd[4314]:      0> 2017-07-20 13:52:26.505919 7feab78f9700 -1 /build/ceph-12.1.1/src/osd/osd_types.h: In function 'static void osd_reqid_t::_denc_finish(ceph::buffer::ptr::iterator&, __u8*, __u8*, char**, uint32_t*)' thread 7feab78f9700 time 2017-07-20 13:52:26.501284
Jul 20 13:52:26 osd1 ceph-osd[4314]: /build/ceph-12.1.1/src/osd/osd_types.h: 117: FAILED assert(pos <= end)
Jul 20 13:52:26 osd1 ceph-osd[4314]:  ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
Jul 20 13:52:26 osd1 ceph-osd[4314]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x5565a2421b72]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  2: (std::enable_if<denc_traits<osd_reqid_t, void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous, void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x5565a1f345d9]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  3: (MOSDRepOp::decode_payload()+0x9e) [0x5565a213c1ce]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  4: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8) [0x5565a245c5d8]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  5: (AsyncConnection::process()+0x1d4e) [0x5565a2678bde]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  6: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x5565a24c3148]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  7: (()+0xb0d0d8) [0x5565a24c70d8]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  8: (()+0xb8c80) [0x7feabad80c80]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  9: (()+0x76ba) [0x7feabb46f6ba]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  10: (clone()+0x6d) [0x7feaba4e63dd]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jul 20 13:52:26 osd1 ceph-osd[4314]: *** Caught signal (Aborted) **
Jul 20 13:52:26 osd1 ceph-osd[4314]:  in thread 7feab78f9700 thread_name:msgr-worker-1
Jul 20 13:52:26 osd1 ceph-osd[4314]:  ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
Jul 20 13:52:26 osd1 ceph-osd[4314]:  1: (()+0xa257a4) [0x5565a23df7a4]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  2: (()+0x11390) [0x7feabb479390]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  3: (gsignal()+0x38) [0x7feaba414428]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  4: (abort()+0x16a) [0x7feaba41602a]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x5565a2421cfe]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  6: (std::enable_if<denc_traits<osd_reqid_t, void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous, void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x5565a1f345d9]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  7: (MOSDRepOp::decode_payload()+0x9e) [0x5565a213c1ce]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  8: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8) [0x5565a245c5d8]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  9: (AsyncConnection::process()+0x1d4e) [0x5565a2678bde]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  10: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x5565a24c3148]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  11: (()+0xb0d0d8) [0x5565a24c70d8]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  12: (()+0xb8c80) [0x7feabad80c80]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  13: (()+0x76ba) [0x7feabb46f6ba]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  14: (clone()+0x6d) [0x7feaba4e63dd]
Jul 20 13:52:26 osd1 ceph-osd[4314]: 2017-07-20 13:52:26.554188 7feab78f9700 -1 *** Caught signal (Aborted) **
Jul 20 13:52:26 osd1 ceph-osd[4314]:  in thread 7feab78f9700 thread_name:msgr-worker-1
Jul 20 13:52:26 osd1 ceph-osd[4314]:  ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
Jul 20 13:52:26 osd1 ceph-osd[4314]:  1: (()+0xa257a4) [0x5565a23df7a4]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  2: (()+0x11390) [0x7feabb479390]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  3: (gsignal()+0x38) [0x7feaba414428]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  4: (abort()+0x16a) [0x7feaba41602a]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x5565a2421cfe]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  6: (std::enable_if<denc_traits<osd_reqid_t, void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous, void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x5565a1f345d9]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  7: (MOSDRepOp::decode_payload()+0x9e) [0x5565a213c1ce]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  8: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8) [0x5565a245c5d8]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  9: (AsyncConnection::process()+0x1d4e) [0x5565a2678bde]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  10: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x5565a24c3148]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  11: (()+0xb0d0d8) [0x5565a24c70d8]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  12: (()+0xb8c80) [0x7feabad80c80]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  13: (()+0x76ba) [0x7feabb46f6ba]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  14: (clone()+0x6d) [0x7feaba4e63dd]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jul 20 13:52:26 osd1 ceph-osd[4314]:      0> 2017-07-20 13:52:26.554188 7feab78f9700 -1 *** Caught signal (Aborted) **
Jul 20 13:52:26 osd1 ceph-osd[4314]:  in thread 7feab78f9700 thread_name:msgr-worker-1
Jul 20 13:52:26 osd1 ceph-osd[4314]:  ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)
Jul 20 13:52:26 osd1 ceph-osd[4314]:  1: (()+0xa257a4) [0x5565a23df7a4]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  2: (()+0x11390) [0x7feabb479390]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  3: (gsignal()+0x38) [0x7feaba414428]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  4: (abort()+0x16a) [0x7feaba41602a]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x5565a2421cfe]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  6: (std::enable_if<denc_traits<osd_reqid_t, void>::supported&&denc_traits<osd_reqid_t, void>::need_contiguous, void>::type decode<osd_reqid_t, denc_traits<osd_reqid_t, void> >(osd_reqid_t&, ceph::buffer::list::iterator&)+0x179) [0x5565a1f345d9]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  7: (MOSDRepOp::decode_payload()+0x9e) [0x5565a213c1ce]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  8: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x18a8) [0x5565a245c5d8]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  9: (AsyncConnection::process()+0x1d4e) [0x5565a2678bde]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  10: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa08) [0x5565a24c3148]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  11: (()+0xb0d0d8) [0x5565a24c70d8]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  12: (()+0xb8c80) [0x7feabad80c80]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  13: (()+0x76ba) [0x7feabb46f6ba]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  14: (clone()+0x6d) [0x7feaba4e63dd]
Jul 20 13:52:26 osd1 ceph-osd[4314]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jul 20 13:52:26 osd1 systemd[1]: ceph-osd@3.service: Main process exited, code=killed, status=6/ABRT
Jul 20 13:52:26 osd1 systemd[1]: ceph-osd@3.service: Unit entered failed state.
Jul 20 13:52:26 osd1 systemd[1]: ceph-osd@3.service: Failed with result 'signal'.
Jul 20 13:52:46 osd1 systemd[1]: ceph-osd@3.service: Service hold-off time over, scheduling restart.
Jul 20 13:52:46 osd1 systemd[1]: Stopped Ceph object storage daemon osd.3.
Jul 20 13:52:46 osd1 systemd[1]: Starting Ceph object storage daemon osd.3...
Jul 20 13:52:47 osd1 systemd[1]: Started Ceph object storage daemon osd.3.
Jul 20 13:52:47 osd1 ceph-osd[4406]: starting osd.3 at - osd_data /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux