Hi Arnaud,
On Fri, Feb 11, 2022 at 2:42 PM Arnaud MARTEL
<arnaud.martel@xxxxxxxxxxxxxxxxxxxx> wrote:
>
> Hi,
>
> MDSs are crashing on my production cluster when trying to unlink some files and I need help :-).
> When looking into the log files, I have identified some associated files and I ran a scrub on the parent directory with force,repair,recursive options. No error were detected but the problem persists.
> 'ceph -s" and "ceph health detail" display no error/warning and my main question is: what are my next steps?
>
>
>
> -3> 2022-02-11T08:36:20.647+0000 7fa372dba700 4 mds.0.server handle_client_request client_request(client.3422129:6687 unlink #0x10002191acc/gpt2_L-4_H-768_trained_pre-20_1_checkpoint_24_norm-2_norm-None_temporal-shifting-0_84_hidden-layer-0-1-2-3-4.o459077 2022-02-11T08:36:20.647472+0000 caller_uid=0, caller_gid=0{0,1001,90590,90596,9060
> 2,90610,90619,90620,90627,90636,}) v4
> -2> 2022-02-11T08:36:20.647+0000 7fa36bdac700 5 mds.0.log _submit_thread 9994621415698~1111 : EOpen [metablob 0x10002191acc, 1 dirs], 1 open files
> -1> 2022-02-11T08:36:20.654+0000 7fa372dba700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6/rpm/el8/BUILD/ceph-16.2.6/src/mds/Server.cc: In function 'void Server::_unlink_local(MDRequestRef&, CDentry*, CDentry*)' thread 7fa372dba
> 700 time 2022-02-11T08:36:20.649556+0000
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6/rpm/el8/BUILD/ceph-16.2.6/src/mds/Server.cc: 7503: FAILED ceph_assert(in->first <= straydn->first)
>
> ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable)
> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fa37b7decce]
> 2: /usr/lib64/ceph/libceph-common.so.2(+0x276ee8) [0x7fa37b7deee8]
> 3: (Server::_unlink_local(boost::intrusive_ptr<MDRequestImpl>&, CDentry*, CDentry*)+0x106a) [0x55e4bf43331a]
> 4: (Server::handle_client_unlink(boost::intrusive_ptr<MDRequestImpl>&)+0x4d9) [0x55e4bf437fe9]
> 5: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0xefb) [0x55e4bf44e82b]
> 6: (MDCache::dispatch_request(boost::intrusive_ptr<MDRequestImpl>&)+0x33) [0x55e4bf5044b3]
> 7: (MDSContext::complete(int)+0x56) [0x55e4bf6c0906]
> 8: (MDSCacheObject::finish_waiting(unsigned long, int)+0xce) [0x55e4bf6e26be]
> 9: (Locker::eval_gather(SimpleLock*, bool, bool*, std::vector<MDSContext*, std::allocator<MDSContext*> >*)+0x13d6) [0x55e4bf594f66]
> 10: (Locker::handle_file_lock(ScatterLock*, boost::intrusive_ptr<MLock const> const&)+0xed1) [0x55e4bf5a3241]
> 11: (Locker::handle_lock(boost::intrusive_ptr<MLock const> const&)+0x1b3) [0x55e4bf5a3db3]
> 12: (Locker::dispatch(boost::intrusive_ptr<Message const> const&)+0xb4) [0x55e4bf5a7fe4]
> 13: (MDSRank::handle_message(boost::intrusive_ptr<Message const> const&)+0xbcc) [0x55e4bf3bf38c]
> 14: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x7bb) [0x55e4bf3c19eb]
> 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x55) [0x55e4bf3c1fe5]
> 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x128) [0x55e4bf3b1f28]
> 17: (DispatchQueue::entry()+0x126a) [0x7fa37ba1c4da]
> 18: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa37bacce21]
> 19: /lib64/libpthread.so.0(+0x814a) [0x7fa37a7c514a]
> 20: clone()
You are hitting this bug: https://tracker.ceph.com/issues/38452. It
seems to happen when an inode field gets corrupted.
Would it be possible to set "debug mds = 20", trigger the crash and
share the log? The logs can be huge in size. You may want to share an
uploaded link if you are fine with that.
There is a workaround mentioned in the above tracker, however, that
requires specific commands pertaining to certain metadata objects to
be run.
>
>
> Arnaud
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
--
Cheers,
Venky
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
