here a copy of pastbin. so there are no similar issues out there in the deep internet and so i cant say the root cause: Sep 30 16:24:47 pve04 systemd[1]: Started Ceph metadata server daemon. Sep 30 16:24:47 pve04 ceph-mds[331479]: starting mds.pve04 at Sep 30 16:24:47 pve04 ceph-mds[331479]: 2021-09-30T16:24:47.852+0200 7ffa18edd700 -1 mds.0.openfiles _load_finish: corrupted header/values: void Anchor::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input Sep 30 16:24:50 pve04 ceph-mds[331479]: 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x609 [2,head] v10159 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x608 [2,head] v10166 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x607 [2,head] v10165 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x606 [2,head] v10168 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x605 [2,head] v10164 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x604 [2,head] v10163 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x603 [2,head] v10162 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x602 [2,head] v10161 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x601 [2,head] v10160 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x600 [2,head] v10167 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: ./src/mds/mdstypes.h: In function 'static void dentry_key_t::decode_helper(std::string_view, std::string&, snapid_t&)' thread 7ffa18edd700 time 2021-09-30T16:24:50.874818+0200 Sep 30 16:24:50 pve04 ceph-mds[331479]: ./src/mds/mdstypes.h: 1389: FAILED ceph_assert(i != std::string::npos) Sep 30 16:24:50 pve04 ceph-mds[331479]: ceph version 16.2.6 (1a6b9a05546f335eeeddb460fdc89caadf80ac7a) pacific (stable) Sep 30 16:24:50 pve04 ceph-mds[331479]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x124) [0x7ffa243c11b4] Sep 30 16:24:50 pve04 ceph-mds[331479]: 2: /usr/lib/ceph/libceph-common.so.2(+0x24f33f) [0x7ffa243c133f] Sep 30 16:24:50 pve04 ceph-mds[331479]: 3: (CDir::_omap_fetched(ceph::buffer::v15_2_0::list&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, bool, int)+0x649) [0x55cd214e56c9] Sep 30 16:24:50 pve04 ceph-mds[331479]: 4: (C_IO_Dir_OMAP_Fetched::finish(int)+0x15a) [0x55cd214fdcca] Sep 30 16:24:50 pve04 ceph-mds[331479]: 5: (MDSContext::complete(int)+0x50) [0x55cd2159a0d0] Sep 30 16:24:50 pve04 ceph-mds[331479]: 6: (MDSIOContextBase::complete(int)+0x433) [0x55cd2159a753] Sep 30 16:24:50 pve04 ceph-mds[331479]: 7: (Finisher::finisher_thread_entry()+0x18d) [0x7ffa2445eacd] Sep 30 16:24:50 pve04 ceph-mds[331479]: 8: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7ffa2411eea7] Sep 30 16:24:50 pve04 ceph-mds[331479]: 9: clone() Sep 30 16:24:50 pve04 ceph-mds[331479]: 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 ./src/mds/mdstypes.h: In function 'static void dentry_key_t::decode_helper(std::string_view, std::string&, snapid_t&)' thread 7ffa18edd700 time 2021-09-30T16:24:50.874818+0200 Sep 30 16:24:50 pve04 ceph-mds[331479]: ./src/mds/mdstypes.h: 1389: FAILED ceph_assert(i != std::string::npos) Sep 30 16:24:50 pve04 ceph-mds[331479]: ceph version 16.2.6 (1a6b9a05546f335eeeddb460fdc89caadf80ac7a) pacific (stable) Sep 30 16:24:50 pve04 ceph-mds[331479]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x124) [0x7ffa243c11b4] Sep 30 16:24:50 pve04 ceph-mds[331479]: 2: /usr/lib/ceph/libceph-common.so.2(+0x24f33f) [0x7ffa243c133f] Sep 30 16:24:50 pve04 ceph-mds[331479]: 3: (CDir::_omap_fetched(ceph::buffer::v15_2_0::list&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, bool, int)+0x649) [0x55cd214e56c9] Sep 30 16:24:50 pve04 ceph-mds[331479]: 4: (C_IO_Dir_OMAP_Fetched::finish(int)+0x15a) [0x55cd214fdcca] Sep 30 16:24:50 pve04 ceph-mds[331479]: 5: (MDSContext::complete(int)+0x50) [0x55cd2159a0d0] Sep 30 16:24:50 pve04 ceph-mds[331479]: 6: (MDSIOContextBase::complete(int)+0x433) [0x55cd2159a753] Sep 30 16:24:50 pve04 ceph-mds[331479]: 7: (Finisher::finisher_thread_entry()+0x18d) [0x7ffa2445eacd] Sep 30 16:24:50 pve04 ceph-mds[331479]: 8: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7ffa2411eea7] Sep 30 16:24:50 pve04 ceph-mds[331479]: 9: clone() Sep 30 16:24:50 pve04 ceph-mds[331479]: *** Caught signal (Aborted) ** Sep 30 16:24:50 pve04 ceph-mds[331479]: in thread 7ffa18edd700 thread_name:MR_Finisher Sep 30 16:24:50 pve04 ceph-mds[331479]: ceph version 16.2.6 (1a6b9a05546f335eeeddb460fdc89caadf80ac7a) pacific (stable) Sep 30 16:24:50 pve04 ceph-mds[331479]: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7ffa2412a140] Sep 30 16:24:50 pve04 ceph-mds[331479]: 2: gsignal() Sep 30 16:24:50 pve04 ceph-mds[331479]: 3: abort() Sep 30 16:24:50 pve04 ceph-mds[331479]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x7ffa243c11fe] Sep 30 16:24:50 pve04 ceph-mds[331479]: 5: /usr/lib/ceph/libceph-common.so.2(+0x24f33f) [0x7ffa243c133f] Sep 30 16:24:50 pve04 ceph-mds[331479]: 6: (CDir::_omap_fetched(ceph::buffer::v15_2_0::list&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, bool, int)+0x649) [0x55cd214e56c9] Sep 30 16:24:50 pve04 ceph-mds[331479]: 7: (C_IO_Dir_OMAP_Fetched::finish(int)+0x15a) [0x55cd214fdcca] Sep 30 16:24:50 pve04 ceph-mds[331479]: 8: (MDSContext::complete(int)+0x50) [0x55cd2159a0d0] Sep 30 16:24:50 pve04 ceph-mds[331479]: 9: (MDSIOContextBase::complete(int)+0x433) [0x55cd2159a753] Sep 30 16:24:50 pve04 ceph-mds[331479]: 10: (Finisher::finisher_thread_entry()+0x18d) [0x7ffa2445eacd] Sep 30 16:24:50 pve04 ceph-mds[331479]: 11: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7ffa2411eea7] Sep 30 16:24:50 pve04 ceph-mds[331479]: 12: clone() Sep 30 16:24:50 pve04 ceph-mds[331479]: 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 *** Caught signal (Aborted) ** Sep 30 16:24:50 pve04 ceph-mds[331479]: in thread 7ffa18edd700 thread_name:MR_Finisher Sep 30 16:24:50 pve04 ceph-mds[331479]: ceph version 16.2.6 (1a6b9a05546f335eeeddb460fdc89caadf80ac7a) pacific (stable) Sep 30 16:24:50 pve04 ceph-mds[331479]: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7ffa2412a140] Sep 30 16:24:50 pve04 ceph-mds[331479]: 2: gsignal() Sep 30 16:24:50 pve04 ceph-mds[331479]: 3: abort() Sep 30 16:24:50 pve04 ceph-mds[331479]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x7ffa243c11fe] Sep 30 16:24:50 pve04 ceph-mds[331479]: 5: /usr/lib/ceph/libceph-common.so.2(+0x24f33f) [0x7ffa243c133f] Sep 30 16:24:50 pve04 ceph-mds[331479]: 6: (CDir::_omap_fetched(ceph::buffer::v15_2_0::list&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, bool, int)+0x649) [0x55cd214e56c9] Sep 30 16:24:50 pve04 ceph-mds[331479]: 7: (C_IO_Dir_OMAP_Fetched::finish(int)+0x15a) [0x55cd214fdcca] Sep 30 16:24:50 pve04 ceph-mds[331479]: 8: (MDSContext::complete(int)+0x50) [0x55cd2159a0d0] Sep 30 16:24:50 pve04 ceph-mds[331479]: 9: (MDSIOContextBase::complete(int)+0x433) [0x55cd2159a753] Sep 30 16:24:50 pve04 ceph-mds[331479]: 10: (Finisher::finisher_thread_entry()+0x18d) [0x7ffa2445eacd] Sep 30 16:24:50 pve04 ceph-mds[331479]: 11: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7ffa2411eea7] Sep 30 16:24:50 pve04 ceph-mds[331479]: 12: clone() Sep 30 16:24:50 pve04 ceph-mds[331479]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Sep 30 16:24:50 pve04 ceph-mds[331479]: -1151> 2021-09-30T16:24:47.852+0200 7ffa18edd700 -1 mds.0.openfiles _load_finish: corrupted header/values: void Anchor::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input Sep 30 16:24:50 pve04 ceph-mds[331479]: -20> 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x609 [2,head] v10159 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: -18> 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x608 [2,head] v10166 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: -16> 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x607 [2,head] v10165 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: -14> 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x606 [2,head] v10168 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: -12> 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x605 [2,head] v10164 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: -10> 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x604 [2,head] v10163 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: -8> 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x603 [2,head] v10162 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: -6> 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x602 [2,head] v10161 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: -4> 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x601 [2,head] v10160 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: -2> 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x600 [2,head] v10167 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: -1> 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 ./src/mds/mdstypes.h: In function 'static void dentry_key_t::decode_helper(std::string_view, std::string&, snapid_t&)' thread 7ffa18edd700 time 2021-09-30T16:24:50.874818+0200 Sep 30 16:24:50 pve04 ceph-mds[331479]: ./src/mds/mdstypes.h: 1389: FAILED ceph_assert(i != std::string::npos) Sep 30 16:24:50 pve04 ceph-mds[331479]: ceph version 16.2.6 (1a6b9a05546f335eeeddb460fdc89caadf80ac7a) pacific (stable) Sep 30 16:24:50 pve04 ceph-mds[331479]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x124) [0x7ffa243c11b4] Sep 30 16:24:50 pve04 ceph-mds[331479]: 2: /usr/lib/ceph/libceph-common.so.2(+0x24f33f) [0x7ffa243c133f] Sep 30 16:24:50 pve04 ceph-mds[331479]: 3: (CDir::_omap_fetched(ceph::buffer::v15_2_0::list&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, bool, int)+0x649) [0x55cd214e56c9] Sep 30 16:24:50 pve04 ceph-mds[331479]: 4: (C_IO_Dir_OMAP_Fetched::finish(int)+0x15a) [0x55cd214fdcca] Sep 30 16:24:50 pve04 ceph-mds[331479]: 5: (MDSContext::complete(int)+0x50) [0x55cd2159a0d0] Sep 30 16:24:50 pve04 ceph-mds[331479]: 6: (MDSIOContextBase::complete(int)+0x433) [0x55cd2159a753] Sep 30 16:24:50 pve04 ceph-mds[331479]: 7: (Finisher::finisher_thread_entry()+0x18d) [0x7ffa2445eacd] Sep 30 16:24:50 pve04 ceph-mds[331479]: 8: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7ffa2411eea7] Sep 30 16:24:50 pve04 ceph-mds[331479]: 9: clone() Sep 30 16:24:50 pve04 ceph-mds[331479]: 0> 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 *** Caught signal (Aborted) ** Sep 30 16:24:50 pve04 ceph-mds[331479]: in thread 7ffa18edd700 thread_name:MR_Finisher Sep 30 16:24:50 pve04 ceph-mds[331479]: ceph version 16.2.6 (1a6b9a05546f335eeeddb460fdc89caadf80ac7a) pacific (stable) Sep 30 16:24:50 pve04 ceph-mds[331479]: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7ffa2412a140] Sep 30 16:24:50 pve04 ceph-mds[331479]: 2: gsignal() Sep 30 16:24:50 pve04 ceph-mds[331479]: 3: abort() Sep 30 16:24:50 pve04 ceph-mds[331479]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x7ffa243c11fe] Sep 30 16:24:50 pve04 ceph-mds[331479]: 5: /usr/lib/ceph/libceph-common.so.2(+0x24f33f) [0x7ffa243c133f] Sep 30 16:24:50 pve04 ceph-mds[331479]: 6: (CDir::_omap_fetched(ceph::buffer::v15_2_0::list&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, bool, int)+0x649) [0x55cd214e56c9] Sep 30 16:24:50 pve04 ceph-mds[331479]: 7: (C_IO_Dir_OMAP_Fetched::finish(int)+0x15a) [0x55cd214fdcca] Sep 30 16:24:50 pve04 ceph-mds[331479]: 8: (MDSContext::complete(int)+0x50) [0x55cd2159a0d0] Sep 30 16:24:50 pve04 ceph-mds[331479]: 9: (MDSIOContextBase::complete(int)+0x433) [0x55cd2159a753] Sep 30 16:24:50 pve04 ceph-mds[331479]: 10: (Finisher::finisher_thread_entry()+0x18d) [0x7ffa2445eacd] Sep 30 16:24:50 pve04 ceph-mds[331479]: 11: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7ffa2411eea7] Sep 30 16:24:50 pve04 ceph-mds[331479]: 12: clone() Sep 30 16:24:50 pve04 ceph-mds[331479]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Sep 30 16:24:50 pve04 ceph-mds[331479]: -1151> 2021-09-30T16:24:47.852+0200 7ffa18edd700 -1 mds.0.openfiles _load_finish: corrupted header/values: void Anchor::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input Sep 30 16:24:50 pve04 ceph-mds[331479]: -20> 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x609 [2,head] v10159 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: -18> 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x608 [2,head] v10166 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: -16> 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x607 [2,head] v10165 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: -14> 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x606 [2,head] v10168 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: -12> 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x605 [2,head] v10164 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: -10> 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x604 [2,head] v10163 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: -8> 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x603 [2,head] v10162 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: -6> 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x602 [2,head] v10161 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: -4> 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x601 [2,head] v10160 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: -2> 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x600 [2,head] v10167 at ~mds0/ Sep 30 16:24:50 pve04 ceph-mds[331479]: [1B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: [70B blob data] Sep 30 16:24:50 pve04 ceph-mds[331479]: -1> 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 ./src/mds/mdstypes.h: In function 'static void dentry_key_t::decode_helper(std::string_view, std::string&, snapid_t&)' thread 7ffa18edd700 time 2021-09-30T16:24:50.874818+0200 Sep 30 16:24:50 pve04 ceph-mds[331479]: ./src/mds/mdstypes.h: 1389: FAILED ceph_assert(i != std::string::npos) Sep 30 16:24:50 pve04 ceph-mds[331479]: ceph version 16.2.6 (1a6b9a05546f335eeeddb460fdc89caadf80ac7a) pacific (stable) Sep 30 16:24:50 pve04 ceph-mds[331479]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x124) [0x7ffa243c11b4] Sep 30 16:24:50 pve04 ceph-mds[331479]: 2: /usr/lib/ceph/libceph-common.so.2(+0x24f33f) [0x7ffa243c133f] Sep 30 16:24:50 pve04 ceph-mds[331479]: 3: (CDir::_omap_fetched(ceph::buffer::v15_2_0::list&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, bool, int)+0x649) [0x55cd214e56c9] Sep 30 16:24:50 pve04 ceph-mds[331479]: 4: (C_IO_Dir_OMAP_Fetched::finish(int)+0x15a) [0x55cd214fdcca] Sep 30 16:24:50 pve04 ceph-mds[331479]: 5: (MDSContext::complete(int)+0x50) [0x55cd2159a0d0] Sep 30 16:24:50 pve04 ceph-mds[331479]: 6: (MDSIOContextBase::complete(int)+0x433) [0x55cd2159a753] Sep 30 16:24:50 pve04 ceph-mds[331479]: 7: (Finisher::finisher_thread_entry()+0x18d) [0x7ffa2445eacd] Sep 30 16:24:50 pve04 ceph-mds[331479]: 8: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7ffa2411eea7] Sep 30 16:24:50 pve04 ceph-mds[331479]: 9: clone() Sep 30 16:24:50 pve04 ceph-mds[331479]: 0> 2021-09-30T16:24:50.872+0200 7ffa18edd700 -1 *** Caught signal (Aborted) ** Sep 30 16:24:50 pve04 ceph-mds[331479]: in thread 7ffa18edd700 thread_name:MR_Finisher Sep 30 16:24:50 pve04 ceph-mds[331479]: ceph version 16.2.6 (1a6b9a05546f335eeeddb460fdc89caadf80ac7a) pacific (stable) Sep 30 16:24:50 pve04 ceph-mds[331479]: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7ffa2412a140] Sep 30 16:24:50 pve04 ceph-mds[331479]: 2: gsignal() Sep 30 16:24:50 pve04 ceph-mds[331479]: 3: abort() Sep 30 16:24:50 pve04 ceph-mds[331479]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x7ffa243c11fe] Sep 30 16:24:50 pve04 ceph-mds[331479]: 5: /usr/lib/ceph/libceph-common.so.2(+0x24f33f) [0x7ffa243c133f] Sep 30 16:24:50 pve04 ceph-mds[331479]: 6: (CDir::_omap_fetched(ceph::buffer::v15_2_0::list&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, bool, int)+0x649) [0x55cd214e56c9] Sep 30 16:24:50 pve04 ceph-mds[331479]: 7: (C_IO_Dir_OMAP_Fetched::finish(int)+0x15a) [0x55cd214fdcca] Sep 30 16:24:50 pve04 ceph-mds[331479]: 8: (MDSContext::complete(int)+0x50) [0x55cd2159a0d0] Sep 30 16:24:50 pve04 ceph-mds[331479]: 9: (MDSIOContextBase::complete(int)+0x433) [0x55cd2159a753] Sep 30 16:24:50 pve04 ceph-mds[331479]: 10: (Finisher::finisher_thread_entry()+0x18d) [0x7ffa2445eacd] Sep 30 16:24:50 pve04 ceph-mds[331479]: 11: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7ffa2411eea7] Sep 30 16:24:50 pve04 ceph-mds[331479]: 12: clone() Sep 30 16:24:50 pve04 ceph-mds[331479]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Sep 30 16:24:50 pve04 systemd[1]: ceph-mds@pve04.service: Main process exited, code=killed, status=6/ABRT Sep 30 16:24:50 pve04 systemd[1]: ceph-mds@pve04.service: Failed with result 'signal'. Sep 30 16:24:51 pve04 systemd[1]: ceph-mds@pve04.service: Scheduled restart job, restart counter is at 485. Sep 30 16:24:51 pve04 systemd[1]: Stopped Ceph metadata server daemon. ________________________________ Von: Eugen Block <eblock@xxxxxx> Gesendet: Freitag, 1. Oktober 2021 14:43:01 An: ceph-users@xxxxxxx Betreff: [URL wurde verändert] Re: MDS: corrupted header/values: decode past end of struct encoding: Malformed input Externe E-Mail! Öffnen Sie nur Links oder Anhänge von vertrauenswürdigen Absendern! I can't access the pastebin, did you verify if you hit the same issue as Stefan referenced (https://sis-schwerin.de/externer-link/?href=https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/KQ5A5OWRIUEOJBC7VILBGDIKPQGJQIWN/)? Before deleting or rebuilding anything I would first check what the root cause is. As Stefan said, don't mess with MDS. Zitat von "von Hoesslin, Volker" <Volker.Hoesslin@xxxxxxx>: > is there any chance to fix this? there are some "advanced metadata > repair tools" > (https://sis-schwerin.de/externer-link/?href=https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/) > but i'm not realy sure is it the right way to handle this issue? > > > i have created an "backup" bevor any tries with this command: > > cephfs-journal-tool journal export backup.bin > > maybe can i delete the mds database and recreate it? is this possible? > > volker. > > ________________________________ > Von: von Hoesslin, Volker > Gesendet: Freitag, 1. Oktober 2021 12:18:58 > An: Stefan Kooman; ceph-users@xxxxxxx > Betreff: AW: Re: MDS: corrupted header/values: decode > past end of struct encoding: Malformed input > > > no, aside the update i had not input this command: > > ceph fs set <fs_name> allow_standby_replay false > > just only the lines from the proxmox update path -> > https://sis-schwerin.de/externer-link/?href=https://pve.proxmox.com/wiki/Ceph_Octopus_to_Pacific#Upgrade_all_CephFS_MDS_daemons > > ________________________________ > Von: Stefan Kooman <stefan@xxxxxx> > Gesendet: Freitag, 1. Oktober 2021 11:54:05 > An: von Hoesslin, Volker; ceph-users@xxxxxxx > Betreff: Re: MDS: corrupted header/values: decode past > end of struct encoding: Malformed input > > Externe E-Mail! Öffnen Sie nur Links oder Anhänge von > vertrauenswürdigen Absendern! > On 10/1/21 11:48, von Hoesslin, Volker wrote: >> i can not see what i have done wrong, this are my update steps: > > Have you set allow_standby_replay to false? > > Have you checked there were no more standby-replay daemons (ceph fs dump)? > > Gr. Stefan > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |