On Tue, Jan 29, 2019 at 8:30 PM Ansgar Jazdzewski <a.jazdzewski@xxxxxxxxxxxxxx> wrote: > > Hi, > > we upgraded from 12.2.8 to 13.2.4 (ubuntu 16.04) > > - after the upgrade (~2 hours after the upgrade) the replay-mds keep > crashing so we tryed to restart all MDS than the filesystem was in > 'failed' state and no MDS is in "activ"-state > - we than tryed to downgrade the MDS to 13.2.1 but had no luck. > ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) > mimic (stable), process ceph-mds, pid 6843 > ... > log_channel(cluster) log [ERR] : corrupt sessionmap values: > buffer::malformed_input: void > session_info_t::decode(ceph::buffer::list::iterator&) no longer > understand old encoding version 6 < 7 > > so we "fixed" the filsystem by following the manual > () > > systemctl stop ceph-mds@`hostname -s`.service > cephfs-journal-tool journal export backup.bin > cephfs-journal-tool --help > cephfs-journal-tool --rank=cephfs:all journal export backup.bin > cephfs-journal-tool --rank=cephfs:all event recover_dentries summary > cephfs-journal-tool --rank=cephfs:all journal reset > cephfs-table-tool all reset session > systemctl restart ceph-mds@mds05.service > systemctl stop ceph-mds@`hostname -s`.service > ceph fs set cephfs down true > ceph fs set cephfs down false > cephfs-table-tool all reset session > cephfs-table-tool all reset inode > cephfs-table-tool all reset snap you have reset inode table. it will cause further damage. you should stop using your fs (umount your client), immediately run ' ceph tell mds.xx scrub start / recursive repair' > systemctl restart ceph-mds@mds05.service > less /var/log/ceph/ceph-mds.mds05.log > systemctl stop ceph-mds@`hostname -s`.service > ceph fs reset cephfs --yes-i-really-mean-it > systemctl restart ceph-mds@mds05.service > less /var/log/ceph/ceph-mds.mds05.log > > after the reset we was able to use the cephfs but we still have some errors > > ... > log [ERR] : unmatched rstat rbytes on single dirfrag 0x10002253f4e, > inode has n(v9 rc2019-01-28 14:42:47.371612 b1158 71=8+63), dirfrag > has n(v9 rc2019-01-28 14:42:47.371612 b1004 65=7+58) > log [ERR] : unmatched fragstat size on single dirfrag 0x10002253db6, > inode has f(v0 m2019-01-28 14:46:47.983292 59=0+59), dirfrag has f(v0 > m2019-01-28 14:46:47.983292 58=0+58) > log [ERR] : unmatched rstat rbytes on single dirfrag 0x10002253db6, > inode has n(v11 rc2019-01-28 14:46:47.983292 b1478 71=11+60), dirfrag > has n(v11 rc2019-01-28 14:46:47.983292 b1347 68=10+58) > ... > > any help is welcome, > Ansgar > > Am Di., 29. Jan. 2019 um 12:32 Uhr schrieb Yan, Zheng <ukernel@xxxxxxxxx>: > > > > upgraded from which version? have you try downgrade ceph-mds to old version? > > > > > > On Mon, Jan 28, 2019 at 9:20 PM Ansgar Jazdzewski > > <a.jazdzewski@xxxxxxxxxxxxxx> wrote: > > > > > > hi folks we need some help with our cephfs, all mds keep crashing > > > > > > starting mds.mds02 at - > > > terminate called after throwing an instance of > > > 'ceph::buffer::bad_alloc' > > > what(): buffer::bad_alloc > > > *** Caught signal (Aborted) ** > > > in thread 7f542d825700 thread_name:md_log_replay > > > ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable) > > > 1: /usr/bin/ceph-mds() [0x7cc8a0] > > > 2: (()+0x11390) [0x7f543cf29390] > > > 3: (gsignal()+0x38) [0x7f543c676428] > > > 4: (abort()+0x16a) [0x7f543c67802a] > > > 5: (__gnu_cxx::__verbose_terminate_handler()+0x135) [0x7f543dae6e65] > > > 6: (__cxxabiv1::__terminate(void (*)())+0x6) [0x7f543dadae46] > > > 7: (()+0x734e91) [0x7f543dadae91] > > > 8: (()+0x7410a4) [0x7f543dae70a4] > > > 9: (ceph::buffer::create_aligned_in_mempool(unsigned int, unsigned > > > int, int)+0x258) [0x7f543d63b348] > > > 10: (ceph::buffer::list::iterator_impl<false>::copy_shallow(unsigned > > > int, ceph::buffer::ptr&)+0xa2) [0x7f543d640ee2] > > > 11: (compact_map_base<std::__cxx11::basic_string<char, > > > std::char_traits<char>, > > > mempool::pool_allocator<(mempool::pool_index_t)18, char> >, > > > ceph::buffer::ptr, std::map<std::__cxx11::basic_string<char, > > > std::char_traits<char>, > > > mempool::pool_allocator<(mempool::pool_index_t)18, char> >, > > > ceph::buffer::ptr, std::less<std::__cxx11::basic_string<char, > > > std::char_traits<char>, mempool::po > > > ol_allocator<(mempool::pool_index_t)18, char> > >, > > > mempool::pool_allocator<(mempool::pool_index_t)18, > > > std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, > > > mempool::pool_allocator<(mempool::pool_index_t)18, char> > const, > > > ceph::buffer::ptr> > > >::decode(ceph::buffer::list::iterator&)+0x122) > > > [0x66b202] > > > 12: (EMetaBlob::fullbit::decode(ceph::buffer::list::iterator&)+0xe3) [0x7aa633] > > > 13: /usr/bin/ceph-mds() [0x7aeae6] > > > 14: (EMetaBlob::replay(MDSRank*, LogSegment*, MDSlaveUpdate*)+0x3d36) [0x7b4fa6] > > > 15: (EImportStart::replay(MDSRank*)+0x5b) [0x7bbb1b] > > > 16: (MDLog::_replay_thread()+0x864) [0x760024] > > > 17: (MDLog::ReplayThread::entry()+0xd) [0x4f487d] > > > 18: (()+0x76ba) [0x7f543cf1f6ba] > > > 19: (clone()+0x6d) [0x7f543c74841d] > > > 2019-01-28 13:10:02.202 7f542d825700 -1 *** Caught signal (Aborted) ** > > > in thread 7f542d825700 thread_name:md_log_replay > > > > > > ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable) > > > 1: /usr/bin/ceph-mds() [0x7cc8a0] > > > 2: (()+0x11390) [0x7f543cf29390] > > > 3: (gsignal()+0x38) [0x7f543c676428] > > > 4: (abort()+0x16a) [0x7f543c67802a] > > > 5: (__gnu_cxx::__verbose_terminate_handler()+0x135) [0x7f543dae6e65] > > > 6: (__cxxabiv1::__terminate(void (*)())+0x6) [0x7f543dadae46] > > > 7: (()+0x734e91) [0x7f543dadae91] > > > 8: (()+0x7410a4) [0x7f543dae70a4] > > > 9: (ceph::buffer::create_aligned_in_mempool(unsigned int, unsigned > > > int, int)+0x258) [0x7f543d63b348] > > > 10: (ceph::buffer::list::iterator_impl<false>::copy_shallow(unsigned > > > int, ceph::buffer::ptr&)+0xa2) [0x7f543d640ee2] > > > 11: (compact_map_base<std::__cxx11::basic_string<char, > > > std::char_traits<char>, > > > mempool::pool_allocator<(mempool::pool_index_t)18, char> >, > > > ceph::buffer::ptr, std::map<std::__cxx11::basic_string<char, > > > std::char_traits<char>, > > > mempool::pool_allocator<(mempool::pool_index_t)18, char> >, > > > ceph::buffer::ptr, std::less<std::__cxx11::basic_string<char, > > > std::char_traits<char>, mempool::po > > > ol_allocator<(mempool::pool_index_t)18, char> > >, > > > mempool::pool_allocator<(mempool::pool_index_t)18, > > > std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, > > > mempool::pool_allocator<(mempool::pool_index_t)18, char> > const, > > > ceph::buffer::ptr> > > >::decode(ceph::buffer::list::iterator&)+0x122) > > > [0x66b202] > > > 12: (EMetaBlob::fullbit::decode(ceph::buffer::list::iterator&)+0xe3) [0x7aa633] > > > 13: /usr/bin/ceph-mds() [0x7aeae6] > > > 14: (EMetaBlob::replay(MDSRank*, LogSegment*, MDSlaveUpdate*)+0x3d36) [0x7b4fa6] > > > 15: (EImportStart::replay(MDSRank*)+0x5b) [0x7bbb1b] > > > 16: (MDLog::_replay_thread()+0x864) [0x760024] > > > 17: (MDLog::ReplayThread::entry()+0xd) [0x4f487d] > > > 18: (()+0x76ba) [0x7f543cf1f6ba] > > > 19: (clone()+0x6d) [0x7f543c74841d] > > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > > > needed to interpret this. > > > > > > 0> 2019-01-28 13:10:02.202 7f542d825700 -1 *** Caught signal > > > (Aborted) ** > > > in thread 7f542d825700 thread_name:md_log_replay > > > > > > ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable) > > > 1: /usr/bin/ceph-mds() [0x7cc8a0] > > > 2: (()+0x11390) [0x7f543cf29390] > > > 3: (gsignal()+0x38) [0x7f543c676428] > > > 4: (abort()+0x16a) [0x7f543c67802a] > > > 5: (__gnu_cxx::__verbose_terminate_handler()+0x135) [0x7f543dae6e65] > > > 6: (__cxxabiv1::__terminate(void (*)())+0x6) [0x7f543dadae46] > > > 7: (()+0x734e91) [0x7f543dadae91] > > > 8: (()+0x7410a4) [0x7f543dae70a4] > > > 9: (ceph::buffer::create_aligned_in_mempool(unsigned int, unsigned > > > int, int)+0x258) [0x7f543d63b348] > > > 10: (ceph::buffer::list::iterator_impl<false>::copy_shallow(unsigned > > > int, ceph::buffer::ptr&)+0xa2) [0x7f543d640ee2] > > > 11: (compact_map_base<std::__cxx11::basic_string<char, > > > std::char_traits<char>, > > > mempool::pool_allocator<(mempool::pool_index_t)18, char> >, > > > ceph::buffer::ptr, std::map<std::__cxx11::basic_string<char, > > > std::char_traits<char>, > > > mempool::pool_allocator<(mempool::pool_index_t)18, char> >, > > > ceph::buffer::ptr, std::less<std::__cxx11::basic_string<char, > > > std::char_traits<char>, mempool::po > > > ol_allocator<(mempool::pool_index_t)18, char> > >, > > > mempool::pool_allocator<(mempool::pool_index_t)18, > > > std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, > > > mempool::pool_allocator<(mempool::pool_index_t)18, char> > const, > > > ceph::buffer::ptr> > > >::decode(ceph::buffer::list::iterator&)+0x122) > > > [0x66b202] > > > 12: (EMetaBlob::fullbit::decode(ceph::buffer::list::iterator&)+0xe3) [0x7aa633] > > > 13: /usr/bin/ceph-mds() [0x7aeae6] > > > 14: (EMetaBlob::replay(MDSRank*, LogSegment*, MDSlaveUpdate*)+0x3d36) [0x7b4fa6] > > > 15: (EImportStart::replay(MDSRank*)+0x5b) [0x7bbb1b] > > > 16: (MDLog::_replay_thread()+0x864) [0x760024] > > > 17: (MDLog::ReplayThread::entry()+0xd) [0x4f487d] > > > 18: (()+0x76ba) [0x7f543cf1f6ba] > > > 19: (clone()+0x6d) [0x7f543c74841d] > > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > > > needed to interpret this. > > > > > > Aborted > > > _______________________________________________ > > > ceph-users mailing list > > > ceph-users@xxxxxxxxxxxxxx > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |