Thanks, i’ll create an issue tracker > 在 2018年3月30日，上午8:19，Yan, Zheng <ukernel@xxxxxxxxx> 写道： > > On Thu, Mar 29, 2018 at 11:55 PM, 陶冬冬 <tdd21151186@xxxxxxxxx> wrote: >> >> I have verified that by adding some debug log, and found it’s truly caused by long wait in mqueue for rejoin. >> but, it seems this kind laggy does no harm ? >> > > It's harmful. I think we should move long work into worker thread. So > it does not block message dispatch > > >>> 在 2018年3月29日，上午10:08，Yan, Zheng <ukernel@xxxxxxxxx> 写道： >>> >>> #2 0x00005555557134eb in MDCache::process_imported_caps >>> (this=this@entry=0x5555566a9000) at >>> /home/zhyan/Ceph/ceph-2/src/mds/MDCache.cc:5397 >>> >>> #3 0x00005555557149a3 in MDCache::rejoin_start >>> (this=this@entry=0x5555566a9000, rejoin_done_=0x5555566b5c80) at >>> /home/zhyan/Ceph/ceph-2/src/mds/MDCache.cc:3950 >>> >>> #4 0x0000555555663f93 in MDSRank::rejoin_start >>> (this=this@entry=0x5555566a8000) at >>> /home/zhyan/Ceph/ceph-2/src/mds/MDSRank.cc:1425 >>> >>> #5 0x00005555556747e3 in MDSRankDispatcher::handle_mds_map >>> (this=0x5555566a8000, m=m@entry=0x555556527400, >>> oldmap=oldmap@entry=0x5555564b2dc0) at >>> /home/zhyan/Ceph/ceph-2/src/mds/MDSRank.cc:1683 >>> >>> #6 0x0000555555655a1e in MDSDaemon::handle_mds_map >>> (this=this@entry=0x55555651e000, m=m@entry=0x555556527400) at >>> /home/zhyan/Ceph/ceph-2/src/mds/MDSDaemon.cc:1000 >>> >>> #7 0x00005555556582fc in MDSDaemon::handle_core_message >>> (this=this@entry=0x55555651e000, m=m@entry=0x555556527400) at >>> /home/zhyan/Ceph/ceph-2/src/mds/MDSDaemon.cc:1204 >>> >>> #8 0x0000555555658aa3 in MDSDaemon::ms_dispatch (this=0x55555651e000, >>> m=0x555556527400) at /home/zhyan/Ceph/ceph-2/src/mds/MDSDaemon.cc:1158 >>> >>> #9 0x00007fffef283eda in Messenger::ms_deliver_dispatch >>> (m=0x555556527400, this=0x55555651a000) at >>> /home/zhyan/Ceph/ceph-2/src/msg/Messenger.h:667 >>> >>> #10 DispatchQueue::entry (this=0x55555651a1a8) at >>> /home/zhyan/Ceph/ceph-2/src/msg/DispatchQueue.cc:201 >>> >>> #11 0x00007fffef31dcfd in DispatchQueue::DispatchThread::entry >>> (this=<optimized out>) at >>> /home/zhyan/Ceph/ceph-2/src/msg/DispatchQueue.h:101 >>> >>> #12 0x00007fffed4f336d in start_thread () from /lib64/libpthread.so.0 >>> >>> #13 0x00007fffec547b4f in clone () from /lib64/libc.so.6 >>> >>> >>> I think the real cause is that MDCache::process_imported_caps() takes >>> too long. prevent messenger from dispatch other message. >>> >>> On Thu, Mar 29, 2018 at 9:37 AM, Yan, Zheng <ukernel@xxxxxxxxx> wrote: >>>> On Thu, Mar 29, 2018 at 9:09 AM, 陶冬冬 <tdd21151186@xxxxxxxxx> wrote: >>>>> Patrick, we are using 32G memory for the mds. >>>>> Zheng, calling mds->heartbeat_reset() could make the healthy check pass so >>>>> that monitor won’t kick it out. >>>>> more frustrating me is about the laggy issue, from the monitor log, i can >>>>> actually see the MDSBeacon are sent without delay. >>>>> but about 50 seconds later, mds start handling that MDSBeacon message. >>>>> so i’m wondering would that possible the message stayed in the mqueue for >>>>> that long time (if the previous message is MDSMap with rejoin state, ant >>>>> that rejoin take long time) >>>>> (meantime, i will do some more investigating about this issue) >>>>> >>>> >>>> If it's really caused by long wait in mqueue. we should limit >>>> concurrent open_ino >>>> started by MDCache::process_imported_caps() >>>> >>>> >>>>> 在 2018年3月29日，上午8:10，Yan, Zheng <ukernel@xxxxxxxxx> 写道： >>>>> >>>>> On Wed, Mar 28, 2018 at 11:14 PM, 陶冬冬 <tdd21151186@xxxxxxxxx> wrote: >>>>> >>>>> Hi Zheng & Patrick, >>>>> >>>>> we are using v12.2.2. >>>>> Recently we’ve met an mds laggy issue (significantly, about 50 seconds) >>>>> i’ve traced the monitor and mds log and found that the MMDSBeacon message >>>>> was actually sent to mds 50 seconds ago. >>>>> so, looks like monitor isn’t laggy , and more worse is that i also found >>>>> that the mds’s health check is failed and eventually monitor >>>>> just kicked out this mds and make it respawn. >>>>> by the way, this happened at rejoin phase. >>>>> >>>>> Following is my analysis : >>>>> The mds health check failure is because the mds tick thread could not get >>>>> the mds_lock due to rejoin. (i found rejoin has many missing ino needed to >>>>> fetch) >>>>> and this leads the mqueue of the DispatchQueue consumed by Dispatcher got >>>>> very slow, eventually make MMDSBeacon in mqueue got dispatched after a big >>>>> delay. >>>>> >>>>> >>>>> how about calling mds->heartbeat_reset() in the loop that fetch inodes >>>>> >>>>> >>>>> >>>>> i want to know if my analysis make sense to you ? if so, i’m wondering can >>>>> we make MMDSBeacon fast dispatch. >>>>> >>>>> Regards, >>>>> Dongdong >>>>> >>>>> >> -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
