Hi Dietmar,
thanks for that info. I repeated a benchmark test we were using when trying to find out what the problem is. Its un-taring an anaconda2 archive, which produces a high mixed load on a file system. I remember that it used to take ca. 4 minutes on a freshly mounted client. After reducing the cache value, it takes only 2 minutes.
The reason we used this as a benchmark is, that a user observed that the extraction stalls after a while. Back then we couldn't reliably reproduce the problem, so we didn't really know what to look for. It was really strange. When I did it as myself, it was fast. When I did it as the other user on the same server, it stalled. It usually happened around the same place when hard-links were created. The ceph documentation mentioned that hard links are a challenge and we didn't have time to dig more into it.
I don't have any observations about possible slow-downs as seen before, but its only a day in production for now. I hope the reduction doesn't have any negative side effects. So far, I didn't receive any complaints.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Dietmar Rieder <dietmar.rieder@xxxxxxxxxxx>
Sent: 20 January 2021 14:18:19
To: Frank Schilder; ceph-users@xxxxxxx
Subject: Re: Re: cephfs: massive drop in MDS requests per second with increasing number of caps
Hi Frank,
yes I ran several tests going down from the default 1M, 256k, 128k, 96k
to 64k which seemed to be opimal in our case.
~Best
Dietmar
On 1/20/21 1:01 PM, Frank Schilder wrote:
> Hi Dietmar,
>
> thanks for that. I reduced the value and, indeed, the number of caps clients were holding started going down.
>
> A question about the particular value of 64K. Did you run several tests and found this one to be optimal, or was it just a lucky guess?
>
> Thanks and best regards,
> =================
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> ________________________________________
> From: Dietmar Rieder <dietmar.rieder@xxxxxxxxxxx>
> Sent: 19 January 2021 13:24:15
> To: Frank Schilder; ceph-users@xxxxxxx
> Subject: Re: Re: cephfs: massive drop in MDS requests per second with increasing number of caps
>
> Hi Frank,
>
> you don't need to remount the fs. The kernel driver should react to the
> change on the MDS.
>
> Best
> Dietmar
>
> On 1/19/21 9:07 AM, Frank Schilder wrote:
>> Hi Dietmar,
>>
>> thanks for discovering this. I also observed in the past that clients can become unbearably slow for no apparent reason. I never managed to reproduce this and, therefore, didn't report it here.
>>
>> A question about setting these flags on an existing mount. Will a "mount -o remount /mnt/cephfs" update client settings from the cluster without interrupting I/O? I couldn't find anything regarding updating config settings in the manual pages.
>>
>> I would be most interested in further updates in this matter and also if you find other flags with positive performance impact.
>>
>> Best regards,
>> =================
>> Frank Schilder
>> AIT Risø Campus
>> Bygning 109, rum S14
>>
>> ________________________________________
>> From: Dietmar Rieder <dietmar.rieder@xxxxxxxxxxx>
>> Sent: 18 January 2021 21:01:50
>> To: ceph-users@xxxxxxx
>> Subject: Re: cephfs: massive drop in MDS requests per second with increasing number of caps
>>
>> Hi Burkhard, hi list,
>>
>>
>> I checked the 'mds_max_caps_per_client' setting and it turned out that
>> it was set to the default value of 1 million. The
>> 'mds_cache_memory_limit' setting, however I had previously set to 40GB.
>>
>>
>> Given this, I now started to play around with the max_caps and set
>> 'mds_max_caps_per_client' 64k:
>>
>> # ceph config set mds mds_max_caps_per_client 65536
>>
>> And this resulted in to a much better an stable performance of ~1.4k
>> req/seq from one client and ~2.9k req/sec when running 2 clients in
>> parallel. Remember it was max ~660 req/sec before using the 1M default
>> and it gradually decreased to ~60 req/sec after some minute never
>> getting higher again unless manually dropping the dentries and inodes
>> from the vm cache on the client. (I guess this is because only 5000 caps
>> are recalled after reaching the mds_max_caps_per_client)
>>
>> I'll keep this for now and observe if it has any impact on other
>> operations or situations.
>>
>> Still I wonder why a higher number (i.e. >64k) of caps on the client
>> destroys the performance completely.
>>
>> Thanks again
>> Dietmar
>>
>> On 1/18/21 6:20 PM, Dietmar Rieder wrote:
>>> Hi Burkhard,
>>>
>>> thanks so much for the quick reply and the explanation and suggestions.
>>> I'll check these settings and eventually change them and report back.
>>>
>>> Best
>>> Dietmar
>>>
>>> On 1/18/21 6:00 PM, Burkhard Linke wrote:
>>>> Hi,
>>>>
>>>> On 1/18/21 5:46 PM, Dietmar Rieder wrote:
>>>>> Hi all,
>>>>>
>>>>> we noticed a massive drop in requests per second a cephfs client is
>>>>> able to perform when we do a recursive chown over a directory with
>>>>> millions of files. As soon as we see about 170k caps on the MDS, the
>>>>> client performance drops from about 660 reqs/sec to 70 reqs/sec.
>>>>>
>>>>> When we then clear dentries and inodes using "sync; echo 2 >
>>>>> /proc/sys/vm/drop_caches" on the client, the request go up to ~660
>>>>> again just to drop again when reaching about 170k caps.
>>>>>
>>>>> See the attached screenshots.
>>>>>
>>>>> When we stop the chown process for a while and restart it ~25min
>>>>> later again it still performs very slowly and the MDS reqs/sec remain
>>>>> low (~60/sec.). Clearing the cache (dentries and inodes) on the
>>>>> client restores the performance again.
>>>>>
>>>>> When we run the same chown on another client in parallel, it starts
>>>>> again with reasonable good performance (while the first client is
>>>>> poorly performing) but eventually it gets slow again just like the
>>>>> first client.
>>>>>
>>>>> Can someone comment on this and explain it?
>>>>> How can this be solved, so that the performance remains stable?
>>>>
>>>> The MDS has a (soft) limit for number of caps per client. If a clients
>>>> starts to requests more caps, the MDS will ask it to release caps.
>>>> This will add an extra network round trip, thus increasing processing
>>>> time. The setting is 'mds_max_caps_per_client'. The default value is 1
>>>> million caps per client, but maybe this setting was changed in our
>>>> configuration or the overall cap limit for the MDS is restricting it.
>>>>
>>>>
>>>> Since each assigned cap increases the memory consumption of the MDS,
>>>> setting an upper limit helps to control the overall amount of memory
>>>> the MDS is using. So the memory target also affects the number of
>>>> active caps an MDS can manage. You need to adjust both values to your
>>>> use case.
>>>>
>>>>
>>>> I would also recommend to monitor the cap usafe of the MDS, e.g. by
>>>> running 'ceph daemonperf mds.<mds name>' in a shell on the mds server.
>>>> Other methods using the various monitoring interfaces provided by ceph
>>>> are also possible.
>>>>
>>>>
>>>> There are also settings that control how fast a client is releasing
>>>> caps for files; maybe tweaking these settings on the client side may
>>>> also help in your case.
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Burkhard
>>>>
>>>> _______________________________________________
>>>> ceph-users mailing list -- ceph-users@xxxxxxx
>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users@xxxxxxx
>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>>
>>
>>
>> --
>> _________________________________________
>> D i e t m a r R i e d e r, Mag.Dr.
>> Innsbruck Medical University
>> Biocenter - Institute of Bioinformatics
>> Innrain 80, 6020 Innsbruck
>> Phone: +43 512 9003 71402
>> Fax: +43 512 9003 73100
>> Email: dietmar.rieder@xxxxxxxxxxx
>> Web: http://www.icbi.at
>>
>>
>
>
> --
> _________________________________________
> D i e t m a r R i e d e r, Mag.Dr.
> Innsbruck Medical University
> Biocenter - Institute of Bioinformatics
> Innrain 80, 6020 Innsbruck
> Phone: +43 512 9003 71402
> Fax: +43 512 9003 73100
> Email: dietmar.rieder@xxxxxxxxxxx
> Web: http://www.icbi.at
>
>
--
_________________________________________
D i e t m a r R i e d e r, Mag.Dr.
Innsbruck Medical University
Biocenter - Institute of Bioinformatics
Innrain 80, 6020 Innsbruck
Phone: +43 512 9003 71402
Fax: +43 512 9003 73100
Email: dietmar.rieder@xxxxxxxxxxx
Web: http://www.icbi.at
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
