On Thu, May 16, 2019 at 4:10 PM Frank Schilder <frans@xxxxxx> wrote: > > Dear Yan and Stefan, > > thanks for the additional information, it should help reproducing the issue. > > The pdsh command executes a bash script that echoes a few values to stdout. Access should be read-only, however, we still have the FS mounted with atime enabled, so there is probably meta data write and synchronisation per access. Files accessed are ssh auth-keys in .ssh and the shell script. The shell script was located in the home-dir of the user and, following your explanations, to reproduce the issue I will create a directory with many entries and execute a test with the many-clients single-file-read load on it. > try setting mds_bal_split_rd and mds_bal_split_wr to very large value. which prevent mds from splitting hot dirfrag Regards Yan, Zheng > I hope it doesn't take too long. > > Thanks for your input! > > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Yan, Zheng <ukernel@xxxxxxxxx> > Sent: 16 May 2019 09:35 > To: Frank Schilder > Subject: Re: mimic: MDS standby-replay causing blocked ops (MDS bug?) > > On Thu, May 16, 2019 at 2:52 PM Frank Schilder <frans@xxxxxx> wrote: > > > > Dear Yan, > > > > OK, I will try to trigger the problem again and dump the information requested. Since it is not easy to get into this situation and I usually need to resolve it fast (its not a test system), is there anything else worth capturing? > > > > just > > ceph daemon mds.x dump_ops_in_flight > ceph daemon mds.x dump cache /tmp/cachedump.x > > > I will get back as soon as it happened again. > > > > In the meantime, I would be grateful if you could shed some light on the following questions: > > > > - Is there a way to cancel an individual operation in the queue? It is a bit harsh to have to fail an MDS for that. > > no > > > - What is the fragmentdir operation doing in a single MDS setup? I thought this was only relevant if multiple MDS daemons are active on a file system. > > > > It splits large directory to smaller parts. > > > > ================= > > Frank Schilder > > AIT Risø Campus > > Bygning 109, rum S14 > > > > ________________________________________ > > From: Yan, Zheng <ukernel@xxxxxxxxx> > > Sent: 16 May 2019 05:50 > > To: Frank Schilder > > Cc: Stefan Kooman; ceph-users@xxxxxxxxxxxxxx > > Subject: Re: mimic: MDS standby-replay causing blocked ops (MDS bug?) > > > > > [...] > > > This time I captured the MDS ops list (log output does not really contain more info than this list). It contains 12 ops and I will include it here in full length (hope this is acceptable): > > > > > > > Your issues were caused by stuck internal op fragmentdir. Can you > > dump mds cache and send the output to us? _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |