Dear Yan and Stefan, thanks for the additional information, it should help reproducing the issue. The pdsh command executes a bash script that echoes a few values to stdout. Access should be read-only, however, we still have the FS mounted with atime enabled, so there is probably meta data write and synchronisation per access. Files accessed are ssh auth-keys in .ssh and the shell script. The shell script was located in the home-dir of the user and, following your explanations, to reproduce the issue I will create a directory with many entries and execute a test with the many-clients single-file-read load on it. I hope it doesn't take too long. Thanks for your input! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Yan, Zheng <ukernel@xxxxxxxxx> Sent: 16 May 2019 09:35 To: Frank Schilder Subject: Re: mimic: MDS standby-replay causing blocked ops (MDS bug?) On Thu, May 16, 2019 at 2:52 PM Frank Schilder <frans@xxxxxx> wrote: > > Dear Yan, > > OK, I will try to trigger the problem again and dump the information requested. Since it is not easy to get into this situation and I usually need to resolve it fast (its not a test system), is there anything else worth capturing? > just ceph daemon mds.x dump_ops_in_flight ceph daemon mds.x dump cache /tmp/cachedump.x > I will get back as soon as it happened again. > > In the meantime, I would be grateful if you could shed some light on the following questions: > > - Is there a way to cancel an individual operation in the queue? It is a bit harsh to have to fail an MDS for that. no > - What is the fragmentdir operation doing in a single MDS setup? I thought this was only relevant if multiple MDS daemons are active on a file system. > It splits large directory to smaller parts. > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Yan, Zheng <ukernel@xxxxxxxxx> > Sent: 16 May 2019 05:50 > To: Frank Schilder > Cc: Stefan Kooman; ceph-users@xxxxxxxxxxxxxx > Subject: Re: mimic: MDS standby-replay causing blocked ops (MDS bug?) > > > [...] > > This time I captured the MDS ops list (log output does not really contain more info than this list). It contains 12 ops and I will include it here in full length (hope this is acceptable): > > > > Your issues were caused by stuck internal op fragmentdir. Can you > dump mds cache and send the output to us? _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |