Hi,
MDSs are crashing on my production cluster when trying to unlink some files and I need help :-).
When looking into the log files, I have identified some associated files and I ran a scrub on the parent directory with force,repair,recursive options. No error were detected but the problem persists.
'ceph -s" and "ceph health detail" display no error/warning and my main question is: what are my next steps?
-3> 2022-02-11T08:36:20.647+0000 7fa372dba700 4 mds.0.server handle_client_request client_request(client.3422129:6687 unlink #0x10002191acc/gpt2_L-4_H-768_trained_pre-20_1_checkpoint_24_norm-2_norm-None_temporal-shifting-0_84_hidden-layer-0-1-2-3-4.o459077 2022-02-11T08:36:20.647472+0000 caller_uid=0, caller_gid=0{0,1001,90590,90596,9060
2,90610,90619,90620,90627,90636,}) v4
-2> 2022-02-11T08:36:20.647+0000 7fa36bdac700 5 mds.0.log _submit_thread 9994621415698~1111 : EOpen [metablob 0x10002191acc, 1 dirs], 1 open files
-1> 2022-02-11T08:36:20.654+0000 7fa372dba700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6/rpm/el8/BUILD/ceph-16.2.6/src/mds/Server.cc: In function 'void Server::_unlink_local(MDRequestRef&, CDentry*, CDentry*)' thread 7fa372dba
700 time 2022-02-11T08:36:20.649556+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6/rpm/el8/BUILD/ceph-16.2.6/src/mds/Server.cc: 7503: FAILED ceph_assert(in->first <= straydn->first)
ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fa37b7decce]
2: /usr/lib64/ceph/libceph-common.so.2(+0x276ee8) [0x7fa37b7deee8]
3: (Server::_unlink_local(boost::intrusive_ptr<MDRequestImpl>&, CDentry*, CDentry*)+0x106a) [0x55e4bf43331a]
4: (Server::handle_client_unlink(boost::intrusive_ptr<MDRequestImpl>&)+0x4d9) [0x55e4bf437fe9]
5: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0xefb) [0x55e4bf44e82b]
6: (MDCache::dispatch_request(boost::intrusive_ptr<MDRequestImpl>&)+0x33) [0x55e4bf5044b3]
7: (MDSContext::complete(int)+0x56) [0x55e4bf6c0906]
8: (MDSCacheObject::finish_waiting(unsigned long, int)+0xce) [0x55e4bf6e26be]
9: (Locker::eval_gather(SimpleLock*, bool, bool*, std::vector<MDSContext*, std::allocator<MDSContext*> >*)+0x13d6) [0x55e4bf594f66]
10: (Locker::handle_file_lock(ScatterLock*, boost::intrusive_ptr<MLock const> const&)+0xed1) [0x55e4bf5a3241]
11: (Locker::handle_lock(boost::intrusive_ptr<MLock const> const&)+0x1b3) [0x55e4bf5a3db3]
12: (Locker::dispatch(boost::intrusive_ptr<Message const> const&)+0xb4) [0x55e4bf5a7fe4]
13: (MDSRank::handle_message(boost::intrusive_ptr<Message const> const&)+0xbcc) [0x55e4bf3bf38c]
14: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x7bb) [0x55e4bf3c19eb]
15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x55) [0x55e4bf3c1fe5]
16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x128) [0x55e4bf3b1f28]
17: (DispatchQueue::entry()+0x126a) [0x7fa37ba1c4da]
18: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa37bacce21]
19: /lib64/libpthread.so.0(+0x814a) [0x7fa37a7c514a]
20: clone()
Arnaud
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
