On Fri, Jan 25, 2019 at 9:49 PM Jonathan Woytek <woytek@xxxxxxxxxxx> wrote: > > Hi friendly ceph folks. A little while after I got the message asking for some stats, we had a network issue that caused us to take all of our processing offline for a few hours. Since we brought everything back up, I have been unable to duplicate the issues I was seeing. Instead, performance of the file writer has been steady around 1.5k files/minute. Dropping the cache causes performance to suffer. We can only get back to the 1.5k/minute average range by restarting all of the mds daemons (well, specifically, it looks like we can restart the first two or three, but resetting the other three or four doesn't seem to make a difference). > > Now, I'm seeing pretty consistent sets of slow requests logged on the first two mds daemons, stating that the slow request is a "rejoin:client.[clientid]". When I parse the clientid's and look at the client lists on the daemons, the clients correspond to the six swarm hosts running the file writers. I'm attaching a small archive here of the performance metrics Zheng asked me to produce a couple of weeks ago. I'm not sure if they are valid for this particular problem. > > jonathan > Looks like you have 5 active mds. I suspect your issue is related to load balancer. Please try disabling mds load balancer (add "mds_bal_max = 0" to mds section of ceph.conf). and use 'export_pin' to manually pin directories to mds (https://ceph.com/community/new-luminous-cephfs-subtree-pinning/) > > On Wed, Jan 9, 2019 at 9:10 PM Yan, Zheng <ukernel@xxxxxxxxx> wrote: >> >> [...] >> Could you please run following command (for each active mds) when >> operations are fast and when operations are slow >> >> - for i in `seq 10`; do ceph daemon mds.xxx dump_historic_ops > >> mds.xxx.$i; sleep 1; done >> >> Then send the results to us >> >> Regards >> Yan, Zheng > > -- > Jonathan Woytek > http://www.dryrose.com > KB3HOZ > PGP: 462C 5F50 144D 6B09 3B65 FCE8 C1DC DEC4 E8B6 AABC _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |