On Mon, Feb 14, 2022 at 5:33 PM Arnaud MARTEL <arnaud.martel@xxxxxxxxxxxxxxxxxxxx> wrote: > > Hi Venky, > > Thank's a lot for your answer. I needed to reduce the number of running MDS before set debug_mds to 20 but, now, I was able to reproduce the crash and generate the full logfile. > You can download it with the following link: https://mycore.core-cloud.net/index.php/s/hBQA5ym3Jkh1O7l > The logfile is not too big, so I didn't remove any line and you will have all the trace. The crash caused by the unlink is at line 39484... > > I don't known how many files are "invalids" (at least 10-20) nor if there is a way to identify all of them. > > Do you need anything else?? Nope. Thanks for the logs. > > Kind regards > Arnaud > > ----- Mail original ----- > De: "Venky Shankar" <vshankar@xxxxxxxxxx> > À: "arnaud martel" <arnaud.martel@xxxxxxxxxxxxxxxxxxxx> > Cc: "ceph-users" <ceph-users@xxxxxxx> > Envoyé: Vendredi 11 Février 2022 15:03:04 > Objet: Re: MDS crash when unlink file > > Hi Arnaud, > > On Fri, Feb 11, 2022 at 2:42 PM Arnaud MARTEL > <arnaud.martel@xxxxxxxxxxxxxxxxxxxx> wrote: > > > > Hi, > > > > MDSs are crashing on my production cluster when trying to unlink some files and I need help :-). > > When looking into the log files, I have identified some associated files and I ran a scrub on the parent directory with force,repair,recursive options. No error were detected but the problem persists. > > 'ceph -s" and "ceph health detail" display no error/warning and my main question is: what are my next steps? > > > > > > > > -3> 2022-02-11T08:36:20.647+0000 7fa372dba700 4 mds.0.server handle_client_request client_request(client.3422129:6687 unlink #0x10002191acc/gpt2_L-4_H-768_trained_pre-20_1_checkpoint_24_norm-2_norm-None_temporal-shifting-0_84_hidden-layer-0-1-2-3-4.o459077 2022-02-11T08:36:20.647472+0000 caller_uid=0, caller_gid=0{0,1001,90590,90596,9060 > > 2,90610,90619,90620,90627,90636,}) v4 > > -2> 2022-02-11T08:36:20.647+0000 7fa36bdac700 5 mds.0.log _submit_thread 9994621415698~1111 : EOpen [metablob 0x10002191acc, 1 dirs], 1 open files > > -1> 2022-02-11T08:36:20.654+0000 7fa372dba700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6/rpm/el8/BUILD/ceph-16.2.6/src/mds/Server.cc: In function 'void Server::_unlink_local(MDRequestRef&, CDentry*, CDentry*)' thread 7fa372dba > > 700 time 2022-02-11T08:36:20.649556+0000 > > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.6/rpm/el8/BUILD/ceph-16.2.6/src/mds/Server.cc: 7503: FAILED ceph_assert(in->first <= straydn->first) > > > > ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) > > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fa37b7decce] > > 2: /usr/lib64/ceph/libceph-common.so.2(+0x276ee8) [0x7fa37b7deee8] > > 3: (Server::_unlink_local(boost::intrusive_ptr<MDRequestImpl>&, CDentry*, CDentry*)+0x106a) [0x55e4bf43331a] > > 4: (Server::handle_client_unlink(boost::intrusive_ptr<MDRequestImpl>&)+0x4d9) [0x55e4bf437fe9] > > 5: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0xefb) [0x55e4bf44e82b] > > 6: (MDCache::dispatch_request(boost::intrusive_ptr<MDRequestImpl>&)+0x33) [0x55e4bf5044b3] > > 7: (MDSContext::complete(int)+0x56) [0x55e4bf6c0906] > > 8: (MDSCacheObject::finish_waiting(unsigned long, int)+0xce) [0x55e4bf6e26be] > > 9: (Locker::eval_gather(SimpleLock*, bool, bool*, std::vector<MDSContext*, std::allocator<MDSContext*> >*)+0x13d6) [0x55e4bf594f66] > > 10: (Locker::handle_file_lock(ScatterLock*, boost::intrusive_ptr<MLock const> const&)+0xed1) [0x55e4bf5a3241] > > 11: (Locker::handle_lock(boost::intrusive_ptr<MLock const> const&)+0x1b3) [0x55e4bf5a3db3] > > 12: (Locker::dispatch(boost::intrusive_ptr<Message const> const&)+0xb4) [0x55e4bf5a7fe4] > > 13: (MDSRank::handle_message(boost::intrusive_ptr<Message const> const&)+0xbcc) [0x55e4bf3bf38c] > > 14: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x7bb) [0x55e4bf3c19eb] > > 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x55) [0x55e4bf3c1fe5] > > 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x128) [0x55e4bf3b1f28] > > 17: (DispatchQueue::entry()+0x126a) [0x7fa37ba1c4da] > > 18: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa37bacce21] > > 19: /lib64/libpthread.so.0(+0x814a) [0x7fa37a7c514a] > > 20: clone() > > You are hitting this bug: https://tracker.ceph.com/issues/38452. It > seems to happen when an inode field gets corrupted. > > Would it be possible to set "debug mds = 20", trigger the crash and > share the log? The logs can be huge in size. You may want to share an > uploaded link if you are fine with that. > > There is a workaround mentioned in the above tracker, however, that > requires specific commands pertaining to certain metadata objects to > be run. > > > > > > > Arnaud > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > > -- > Cheers, > Venky > -- Cheers, Venky _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |