Re: Sporadic mgr segmentation fault

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Tue, Apr 21, 2020 at 11:39 PM XuYun <yunxu@xxxxxx> wrote:
>
> Dear ceph users,
>
> We are experiencing sporadic mgr crash in all three ceph clusters (version 14.2.6 and version 14.2.8), the crash log is:
>
> 2020-04-17 23:10:08.986 7fed7fe07700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.8/rpm/el7/BUILD/ceph-14.2.8/src/common/buffer.cc: In function 'const char* ceph::buffer::v14_2_0::ptr::c_str() const' thread 7fed7fe07700 time 2020-04-17 23:10:08.984887
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.8/rpm/el7/BUILD/ceph-14.2.8/src/common/buffer.cc: 578: FAILED ceph_assert(_raw)
>
>  ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus (stable)
>  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7fed8605c325]
>  2: (()+0x2534ed) [0x7fed8605c4ed]
>  3: (()+0x5a21ed) [0x7fed863ab1ed]
>  4: (PosixConnectedSocketImpl::send(ceph::buffer::v14_2_0::list&, bool)+0xbd) [0x7fed863840ed]
>  5: (AsyncConnection::_try_send(bool)+0xb6) [0x7fed8632fc76]
>  6: (ProtocolV2::write_message(Message*, bool)+0x832) [0x7fed8635bf52]
>  7: (ProtocolV2::write_event()+0x175) [0x7fed863718c5]
>  8: (AsyncConnection::handle_write()+0x40) [0x7fed86332600]
>  9: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0x1397) [0x7fed8637f997]
>  10: (()+0x57c977) [0x7fed86385977]
>  11: (()+0x80bdaf) [0x7fed86614daf]
>  12: (()+0x7e65) [0x7fed8394ce65]
>  13: (clone()+0x6d) [0x7fed825fa88d]
>
> 2020-04-17 23:10:08.990 7fed7ee05700 -1 *** Caught signal (Segmentation fault) **
>  in thread 7fed7ee05700 thread_name:msgr-worker-2
>
>  ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus (stable)
>  1: (()+0xf5f0) [0x7fed839545f0]
>  2: (ceph::buffer::v14_2_0::ptr::release()+0x8) [0x7fed863aafd8]
>  3: (ceph::crypto::onwire::AES128GCM_OnWireTxHandler::~AES128GCM_OnWireTxHandler()+0x59) [0x7fed86388669]
>  4: (ProtocolV2::reset_recv_state()+0x11f) [0x7fed8635f5af]
>  5: (ProtocolV2::stop()+0x77) [0x7fed8635f857]
>  6: (ProtocolV2::handle_existing_connection(boost::intrusive_ptr<AsyncConnection>)+0x5ef) [0x7fed86374f8f]
>  7: (ProtocolV2::handle_client_ident(ceph::buffer::v14_2_0::list&)+0xd9c) [0x7fed8637673c]
>  8: (ProtocolV2::handle_frame_payload()+0x1fb) [0x7fed86376c1b]
>  9: (ProtocolV2::handle_read_frame_dispatch()+0x150) [0x7fed86376e70]
>  10: (ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&, int)+0x44d) [0x7fed863773cd]
>  11: (ProtocolV2::run_continuation(Ct<ProtocolV2>&)+0x34) [0x7fed86360534]
>  12: (AsyncConnection::process()+0x186) [0x7fed86330656]
>  13: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa15) [0x7fed8637f015]
>  14: (()+0x57c977) [0x7fed86385977]
>  15: (()+0x80bdaf) [0x7fed86614daf]
>  16: (()+0x7e65) [0x7fed8394ce65]
>  17: (clone()+0x6d) [0x7fed825fa88d]
>  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
>
> Any thoughts about this issue?

Looks like https://tracker.ceph.com/issues/42026 which was recently
backported to the Nautilus branch via
https://github.com/ceph/ceph/pull/33820

You could try a build with those patches or wait for 14.2.9

-- 
Cheers,
Brad
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux