Hi all, We have a cephfs cluster in production for about 2 months and, for the past 2-3 weeks, we are regularly experiencing MDS crash loops (every 3-4 hours if we have some user activity). A temporary fix is to remove the MDSs in error (or unknown) state, stop samba & nfs-ganesha gateways, then wipe all sessions. Sometimes, we have to repeat this procedure 2 or 3 times to have our cephfs back and working... When looking in the MDS log files, I noticed that all crashs have the following stack trace: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6/rpm/el8/BUILD/ceph-16.2.6/src/mds/Server.cc: 7503: FAILED ceph_assert(in->first <= straydn->first) ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7eff2644bcce] 2: /usr/lib64/ceph/libceph-common.so.2(+0x276ee8) [0x7eff2644bee8] 3: (Server::_unlink_local(boost::intrusive_ptr<MDRequestImpl>&, CDentry*, CDentry*)+0x106a) [0x559c8f83331a] 4: (Server::handle_client_unlink(boost::intrusive_ptr<MDRequestImpl>&)+0x4d9) [0x559c8f837fe9] 5: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0xefb) [0x559c8f84e82b] 6: (Server::handle_client_request(boost::intrusive_ptr<MClientRequest const> const&)+0x3fc) [0x559c8f859aac] 7: (Server::dispatch(boost::intrusive_ptr<Message const> const&)+0x12b) [0x559c8f86258b] 8: (MDSRank::handle_message(boost::intrusive_ptr<Message const> const&)+0xbb4) [0x559c8f7bf374] 9: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x7bb) [0x559c8f7c19eb] 10: (MDSRank::retry_dispatch(boost::intrusive_ptr<Message const> const&)+0x16) [0x559c8f7c1f86] 11: (MDSContext::complete(int)+0x56) [0x559c8fac0906] 12: (MDSRank::_advance_queues()+0x84) [0x559c8f7c0a54] 13: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x204) [0x559c8f7c1434] 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x55) [0x559c8f7c1fe5] 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x128) [0x559c8f7b1f28] 16: (DispatchQueue::entry()+0x126a) [0x7eff266894da] 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7eff26739e21] 18: /lib64/libpthread.so.0(+0x814a) [0x7eff2543214a] 19: clone() I found an analog case in the ceph tracker website ( [ https://tracker.ceph.com/issues/41147 | https://tracker.ceph.com/issues/41147 ] ) so I suspected an inode corruption and I started a cephfs scrub (ceph tell mds.cephfsvol:0 scrub start / recursive,repair). As we have a lot of files (about 200 millions entries for 200 TB), I don't know how long time it will take nor: - If this will correct the situation - What to do to avoid the same situation in the future Some information about our ceph cluster (pacific 16.2.6 with containers): ********************************************************** ceph -s ********************************************************** cluster: id: 2943b4fe-2063-11ec-a560-e43d1a1bc30f health: HEALTH_WARN 1 MDSs report oversized cache services: mon: 5 daemons, quorum cephp03,cephp06,cephp05,cephp01,cephp02 (age 12d) mgr: cephp01.smfvfd(active, since 12d), standbys: cephp02.equfuj mds: 2/2 daemons up, 4 standby osd: 264 osds: 264 up (since 12d), 264 in (since 9w) rbd-mirror: 1 daemon active (1 hosts) task status: scrub status: mds.cephfsvol.cephp02.wsokro: idle+waiting paths [/] mds.cephfsvol.cephp05.qneike: active paths [/] data: volumes: 1/1 healthy pools: 5 pools, 2176 pgs objects: 595.12M objects, 200 TiB usage: 308 TiB used, 3.3 PiB / 3.6 PiB avail pgs: 2167 active+clean 7 active+clean+scrubbing+deep 2 active+clean+scrubbing io: client: 39 KiB/s rd, 152 KiB/s wr, 27 op/s rd, 27 op/s wr ********************************************************** # ceph fs get cephfsvol ********************************************************** Filesystem 'cephfsvol' (1) fs_name cephfsvol epoch 106554 flags 12 created 2021-09-28T14:19:54.399567+0000 modified 2022-02-08T12:57:00.653514+0000 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 5497558138880 required_client_features {} last_failure 0 last_failure_osd_epoch 41205 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 2 in 0,1 up {0=4044909,1=3325354} failed damaged stopped data_pools [3,4] metadata_pool 2 inline_data disabled balancer standby_count_wanted 1 [mds.cephfsvol.cephp05.qneike{0:4044909} state up:active seq 1789 export targets 1 join_fscid=1 addr [v2:10.2.100.5:6800/2702983829,v1:10.2.100.5:6801/2702983829] compat {c=[1],r=[1],i=[7ff]}] [mds.cephfsvol.cephp02.wsokro{1:32bdaa} state up:active seq 18a02 export targets 0 join_fscid=1 addr [v2:10.2.100.2:1a90/aa660301,v1:10.2.100.2:1a91/aa660301] compat {c=[1],r=[1],i=[7ff]}] [peers=] Any help or advice would be greatly appreciated.... Arnaud _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |