Re: tuning ceph mds cache settings

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Fri, Jan 25, 2019 at 9:49 PM Jonathan Woytek <woytek@xxxxxxxxxxx> wrote:
>
> Hi friendly ceph folks. A little while after I got the message asking for some stats, we had a network issue that caused us to take all of our processing offline for a few hours. Since we brought everything back up, I have been unable to duplicate the issues I was seeing. Instead, performance of the file writer has been steady around 1.5k files/minute. Dropping the cache causes performance to suffer. We can only get back to the 1.5k/minute average range by restarting all of the mds daemons (well, specifically, it looks like we can restart the first two or three, but resetting the other three or four doesn't seem to make a difference).
>
> Now, I'm seeing pretty consistent sets of slow requests logged on the first two mds daemons, stating that the slow request is a "rejoin:client.[clientid]". When I parse the clientid's and look at the client lists on the daemons, the clients correspond to the six swarm hosts running the file writers. I'm attaching a small archive here of the performance metrics Zheng asked me to produce a couple of weeks ago. I'm not sure if they are valid for this particular problem.
>
> jonathan
>

Looks like you have 5 active mds. I suspect your issue is related to
load balancer.  Please try disabling mds load balancer (add
"mds_bal_max = 0" to mds section of ceph.conf). and use 'export_pin'
to manually pin directories to mds
(https://ceph.com/community/new-luminous-cephfs-subtree-pinning/)


>
> On Wed, Jan 9, 2019 at 9:10 PM Yan, Zheng <ukernel@xxxxxxxxx> wrote:
>>
>> [...]
>> Could you please run following command (for each active mds) when
>> operations are fast and when operations are slow
>>
>> - for i in `seq 10`; do ceph daemon mds.xxx dump_historic_ops >
>> mds.xxx.$i; sleep 1; done
>>
>> Then send the results to us
>>
>> Regards
>> Yan, Zheng
>
> --
> Jonathan Woytek
> http://www.dryrose.com
> KB3HOZ
> PGP:  462C 5F50 144D 6B09 3B65  FCE8 C1DC DEC4 E8B6 AABC
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux