upgraded from which version? have you try downgrade ceph-mds to old version? On Mon, Jan 28, 2019 at 9:20 PM Ansgar Jazdzewski <a.jazdzewski@xxxxxxxxxxxxxx> wrote: > > hi folks we need some help with our cephfs, all mds keep crashing > > starting mds.mds02 at - > terminate called after throwing an instance of > 'ceph::buffer::bad_alloc' > what(): buffer::bad_alloc > *** Caught signal (Aborted) ** > in thread 7f542d825700 thread_name:md_log_replay > ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable) > 1: /usr/bin/ceph-mds() [0x7cc8a0] > 2: (()+0x11390) [0x7f543cf29390] > 3: (gsignal()+0x38) [0x7f543c676428] > 4: (abort()+0x16a) [0x7f543c67802a] > 5: (__gnu_cxx::__verbose_terminate_handler()+0x135) [0x7f543dae6e65] > 6: (__cxxabiv1::__terminate(void (*)())+0x6) [0x7f543dadae46] > 7: (()+0x734e91) [0x7f543dadae91] > 8: (()+0x7410a4) [0x7f543dae70a4] > 9: (ceph::buffer::create_aligned_in_mempool(unsigned int, unsigned > int, int)+0x258) [0x7f543d63b348] > 10: (ceph::buffer::list::iterator_impl<false>::copy_shallow(unsigned > int, ceph::buffer::ptr&)+0xa2) [0x7f543d640ee2] > 11: (compact_map_base<std::__cxx11::basic_string<char, > std::char_traits<char>, > mempool::pool_allocator<(mempool::pool_index_t)18, char> >, > ceph::buffer::ptr, std::map<std::__cxx11::basic_string<char, > std::char_traits<char>, > mempool::pool_allocator<(mempool::pool_index_t)18, char> >, > ceph::buffer::ptr, std::less<std::__cxx11::basic_string<char, > std::char_traits<char>, mempool::po > ol_allocator<(mempool::pool_index_t)18, char> > >, > mempool::pool_allocator<(mempool::pool_index_t)18, > std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, > mempool::pool_allocator<(mempool::pool_index_t)18, char> > const, > ceph::buffer::ptr> > > >::decode(ceph::buffer::list::iterator&)+0x122) > [0x66b202] > 12: (EMetaBlob::fullbit::decode(ceph::buffer::list::iterator&)+0xe3) [0x7aa633] > 13: /usr/bin/ceph-mds() [0x7aeae6] > 14: (EMetaBlob::replay(MDSRank*, LogSegment*, MDSlaveUpdate*)+0x3d36) [0x7b4fa6] > 15: (EImportStart::replay(MDSRank*)+0x5b) [0x7bbb1b] > 16: (MDLog::_replay_thread()+0x864) [0x760024] > 17: (MDLog::ReplayThread::entry()+0xd) [0x4f487d] > 18: (()+0x76ba) [0x7f543cf1f6ba] > 19: (clone()+0x6d) [0x7f543c74841d] > 2019-01-28 13:10:02.202 7f542d825700 -1 *** Caught signal (Aborted) ** > in thread 7f542d825700 thread_name:md_log_replay > > ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable) > 1: /usr/bin/ceph-mds() [0x7cc8a0] > 2: (()+0x11390) [0x7f543cf29390] > 3: (gsignal()+0x38) [0x7f543c676428] > 4: (abort()+0x16a) [0x7f543c67802a] > 5: (__gnu_cxx::__verbose_terminate_handler()+0x135) [0x7f543dae6e65] > 6: (__cxxabiv1::__terminate(void (*)())+0x6) [0x7f543dadae46] > 7: (()+0x734e91) [0x7f543dadae91] > 8: (()+0x7410a4) [0x7f543dae70a4] > 9: (ceph::buffer::create_aligned_in_mempool(unsigned int, unsigned > int, int)+0x258) [0x7f543d63b348] > 10: (ceph::buffer::list::iterator_impl<false>::copy_shallow(unsigned > int, ceph::buffer::ptr&)+0xa2) [0x7f543d640ee2] > 11: (compact_map_base<std::__cxx11::basic_string<char, > std::char_traits<char>, > mempool::pool_allocator<(mempool::pool_index_t)18, char> >, > ceph::buffer::ptr, std::map<std::__cxx11::basic_string<char, > std::char_traits<char>, > mempool::pool_allocator<(mempool::pool_index_t)18, char> >, > ceph::buffer::ptr, std::less<std::__cxx11::basic_string<char, > std::char_traits<char>, mempool::po > ol_allocator<(mempool::pool_index_t)18, char> > >, > mempool::pool_allocator<(mempool::pool_index_t)18, > std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, > mempool::pool_allocator<(mempool::pool_index_t)18, char> > const, > ceph::buffer::ptr> > > >::decode(ceph::buffer::list::iterator&)+0x122) > [0x66b202] > 12: (EMetaBlob::fullbit::decode(ceph::buffer::list::iterator&)+0xe3) [0x7aa633] > 13: /usr/bin/ceph-mds() [0x7aeae6] > 14: (EMetaBlob::replay(MDSRank*, LogSegment*, MDSlaveUpdate*)+0x3d36) [0x7b4fa6] > 15: (EImportStart::replay(MDSRank*)+0x5b) [0x7bbb1b] > 16: (MDLog::_replay_thread()+0x864) [0x760024] > 17: (MDLog::ReplayThread::entry()+0xd) [0x4f487d] > 18: (()+0x76ba) [0x7f543cf1f6ba] > 19: (clone()+0x6d) [0x7f543c74841d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this. > > 0> 2019-01-28 13:10:02.202 7f542d825700 -1 *** Caught signal > (Aborted) ** > in thread 7f542d825700 thread_name:md_log_replay > > ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable) > 1: /usr/bin/ceph-mds() [0x7cc8a0] > 2: (()+0x11390) [0x7f543cf29390] > 3: (gsignal()+0x38) [0x7f543c676428] > 4: (abort()+0x16a) [0x7f543c67802a] > 5: (__gnu_cxx::__verbose_terminate_handler()+0x135) [0x7f543dae6e65] > 6: (__cxxabiv1::__terminate(void (*)())+0x6) [0x7f543dadae46] > 7: (()+0x734e91) [0x7f543dadae91] > 8: (()+0x7410a4) [0x7f543dae70a4] > 9: (ceph::buffer::create_aligned_in_mempool(unsigned int, unsigned > int, int)+0x258) [0x7f543d63b348] > 10: (ceph::buffer::list::iterator_impl<false>::copy_shallow(unsigned > int, ceph::buffer::ptr&)+0xa2) [0x7f543d640ee2] > 11: (compact_map_base<std::__cxx11::basic_string<char, > std::char_traits<char>, > mempool::pool_allocator<(mempool::pool_index_t)18, char> >, > ceph::buffer::ptr, std::map<std::__cxx11::basic_string<char, > std::char_traits<char>, > mempool::pool_allocator<(mempool::pool_index_t)18, char> >, > ceph::buffer::ptr, std::less<std::__cxx11::basic_string<char, > std::char_traits<char>, mempool::po > ol_allocator<(mempool::pool_index_t)18, char> > >, > mempool::pool_allocator<(mempool::pool_index_t)18, > std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, > mempool::pool_allocator<(mempool::pool_index_t)18, char> > const, > ceph::buffer::ptr> > > >::decode(ceph::buffer::list::iterator&)+0x122) > [0x66b202] > 12: (EMetaBlob::fullbit::decode(ceph::buffer::list::iterator&)+0xe3) [0x7aa633] > 13: /usr/bin/ceph-mds() [0x7aeae6] > 14: (EMetaBlob::replay(MDSRank*, LogSegment*, MDSlaveUpdate*)+0x3d36) [0x7b4fa6] > 15: (EImportStart::replay(MDSRank*)+0x5b) [0x7bbb1b] > 16: (MDLog::_replay_thread()+0x864) [0x760024] > 17: (MDLog::ReplayThread::entry()+0xd) [0x4f487d] > 18: (()+0x76ba) [0x7f543cf1f6ba] > 19: (clone()+0x6d) [0x7f543c74841d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this. > > Aborted > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |