Hello Eugen,
thank you very much for the full explanation.
This fixed our cluster and I am sure this helps a lot of people around
the world since this is a problem occuring everywhere.
I think this should be added to the documentation:
https://docs.ceph.com/en/latest/cephfs/cache-configuration/#mds-recall
or better:
https://docs.ceph.com/en/quincy/cephfs/health-messages/#mds-client-recall-mds-health-client-recall-many
Best wishes!
Malte
Am 09.08.22 um 16:34 schrieb Eugen Block:
> Hi,
>
>> did you have some success with modifying the mentioned values?
>
> yes, the SUSE team helped identifying the issue, I can share the
> explanation:
>
> ---snip---
> Every second (mds_cache_trim_interval config param) the mds is running
> "cache trim" procedure. One of the steps of this procedure is "recall
> client state". During this step it checks every client (session) if it
> needs to recall caps. There are several criteria for this:
>
> 1) the cache is full (exceeds mds_cache_memory_limit) and needs some
> inodes to be released;
> 2) the client exceeds mds_max_caps_per_client (1M by default);
> 3) the client is inactive.
>
> To determine a client (session) inactivity, the session's cache_liveness
> parameters is checked and compared with the value:
>
> (num_caps >> mds_session_cache_liveness_magnitude)
>
> where mds_session_cache_liveness_magnitude is a config param (10 by
> default).
> If cache_liveness is smaller than this calculated value the session is
> considered inactive and the mds sends "recall caps" request for all
> cached caps (actually the recall value is `num_caps -
> mds_min_caps_per_client(100)`).
>
> And if the client is not releasing the caps fast, the next second it
> repeats again, i.e. the mds will send "recall caps" with high value
> again and so on and the "total" counter of "recall caps" for the session
> will grow, eventually exceeding the mon warning limit.
> There is a throttling mechanism, controlled by
> mds_recall_max_decay_threshold parameter (126K by default), which should
> reduce the rate of "recall caps" counter grow but it looks like it is
> not enough for this case.
>
> From the collected sessions, I see that during that 30 minute period
> the total num_caps for that client decreased by about 3500.
> ...
> Here is an example. A client is having 20k caps cached. At some moment
> the server decides the client is inactive (because the session's
> cache_liveness value is low). It starts to ask the client to release
> caps down to mds_min_caps_per_client value (100 by default). For this
> every seconds it sends recall_caps asking to release `caps_num -
> mds_min_caps_per_client` caps (but not more than `mds_recall_max_caps`,
> which is 30k by default). A client is starting to release, but is
> releasing with a rate e.g. only 100 caps per second.
>
> So in the first second the mds sends recall_caps = 20k - 100
> the second second recall_caps = (20k - 100) - 100
> the third second recall_caps = (20k - 200) - 100
> and so on
>
> And every time it sends recall_caps it updates the session's recall_caps
> value, which is calculated how many recall_caps sent in the last
> minute. I.e. the counter is growing quickly, eventually exceeding
> mds_recall_warning_threshold, which is 128K by default, and ceph starts
> to report "failing to respond to cache pressure" warning in the status.
>
> Now, after we set mds_recall_max_caps to 3K, in this situation the mds
> server sends only 3K recall_caps per second, and the maximum value the
> session's recall_caps value may have (if the mds is sending 3K every
> second for at least one minute) is 60 * 3K = 180K. I.e. it is still
> possible to achieve mds_recall_warning_threshold but only if a client is
> not "responding" for long period, and as your experiments show it is not
> the case.
> ---snip---
>
> So what helped us here was to decrease mds_recall_max_caps in 1k steps,
> starting with 10000. This didn't reduce the warnings so I decreased it
> to 3000 and I haven't seen those warnings since then. Also I decreased
> the mds_cache_memory_limit again, it wasn't helping here.
>
> Regards,
> Eugen
>
>
> Zitat von Malte Stroem <malte.stroem@xxxxxxxxx>:
>
>> Hello Eugen,
>>
>> did you have some success with modifying the mentioned values?
>>
>> Or some others from:
>>
>> https://docs.ceph.com/en/latest/cephfs/cache-configuration/
>>
>> Best,
>> Malte
>>
>> Am 15.06.22 um 14:12 schrieb Eugen Block:
>>> Hi *,
>>>
>>> I finally caught some debug logs during the cache pressure warnings.
>>> In the meantime I had doubled the mds_cache_memory_limit to 128 GB
>>> which decreased the number cache pressure messages significantly, but
>>> they still appear a few times per day.
>>>
>>> Turning on debug logs for a few seconds results in a 1 GB file, but I
>>> found this message:
>>>
>>> 2022-06-15 10:07:34.254 7fdbbd44a700 2 mds.beacon.stmailmds01b-8
>>> Session chead015:cephfs_client (2757628057) is not releasing caps
>>> fast enough. Recalled caps at 390118 > 262144
>>> (mds_recall_warning_threshold).
>>>
>>> So now I know which limit is reached here, the question is what to do
>>> about it? Should I increase the mds_recall_warning_threshold (default
>>> 256k) or should I maybe increase mds_recall_max_caps (currently at
>>> 60k, default is 50k)? Any other suggestions? I'd appreciate any
>>> comments.
>>>
>>> Thanks,
>>> Eugen
>>>
>>>
>>> Zitat von Eugen Block <eblock@xxxxxx>:
>>>
>>>> Hi,
>>>>
>>>> I'm currently debugging a reoccuring issue with multi-active MDS.
>>>> The cluster is still on Nautilus and can't be upgraded at this time.
>>>> There have been many discussions about "cache pressure" and I was
>>>> able to find the right settings a couple of times, but before I
>>>> change too much in this setup I'd like to ask for your opinion. I'll
>>>> add some information at the end.
>>>> So we have 16 active MDS daemons spread over 2 servers for one
>>>> cephfs (8 daemons per server) with mds_cache_memory_limit = 64GB,
>>>> the MDS servers are mostly idle except for some short peaks. Each of
>>>> the MDS daemons uses around 2 GB according to 'ceph daemon mds.<MDS>
>>>> cache status', so we're nowhere near the 64GB limit. There are
>>>> currently 25 servers that mount the cephs as clients.
>>>> Watching the ceph health I can see that the reported clients with
>>>> cache pressure change, so they are not actually stuck but just don't
>>>> respond as quickly as the MDS would like them to (I assume). For
>>>> some of the mentioned clients I see high values for
>>>> .recall_caps.value in the 'daemon session ls' output (at the bottom).
>>>>
>>>> The docs basically state this:
>>>>> When the MDS needs to shrink its cache (to stay within
>>>>> mds_cache_size), it sends messages to clients to shrink their
>>>>> caches too. The client is unresponsive to MDS requests to release
>>>>> cached inodes. Either the client is unresponsive or has a bug
>>>>
>>>> To me it doesn't seem like the MDS servers are near the cache size
>>>> limit, so it has to be the clients, right? In a different setup it
>>>> helped to decrease the client_oc_size from 200MB to 100MB, but then
>>>> there's also client_cache_size with 16K default. I'm not sure what
>>>> the best approach would be here. I'd appreciate any comments on how
>>>> to size the various cache/caps/threshold configurations.
>>>>
>>>> Thanks!
>>>> Eugen
>>>>
>>>>
>>>> ---snip---
>>>> # ceph daemon mds.<MDS> session ls
>>>>
>>>> "id": 2728101146,
>>>> "entity": {
>>>> "name": {
>>>> "type": "client",
>>>> "num": 2728101146
>>>> },
>>>> [...]
>>>> "nonce": 1105499797
>>>> }
>>>> },
>>>> "state": "open",
>>>> "num_leases": 0,
>>>> "num_caps": 16158,
>>>> "request_load_avg": 0,
>>>> "uptime": 1118066.210318422,
>>>> "requests_in_flight": 0,
>>>> "completed_requests": [],
>>>> "reconnecting": false,
>>>> "recall_caps": {
>>>> "value": 788916.8276369586,
>>>> "halflife": 60
>>>> },
>>>> "release_caps": {
>>>> "value": 8.814981576458962,
>>>> "halflife": 60
>>>> },
>>>> "recall_caps_throttle": {
>>>> "value": 27379.27162576508,
>>>> "halflife": 1.5
>>>> },
>>>> "recall_caps_throttle2o": {
>>>> "value": 5382.261925615086,
>>>> "halflife": 0.5
>>>> },
>>>> "session_cache_liveness": {
>>>> "value": 12.91841737465921,
>>>> "halflife": 300
>>>> },
>>>> "cap_acquisition": {
>>>> "value": 0,
>>>> "halflife": 10
>>>> },
>>>> [...]
>>>> "used_inos": [],
>>>> "client_metadata": {
>>>> "features": "0x0000000000003bff",
>>>> "entity_id": "cephfs_client",
>>>>
>>>>
>>>> # ceph fs status
>>>>
>>>> cephfs - 25 clients
>>>> ======
>>>> +------+--------+----------------+---------------+-------+-------+
>>>> | Rank | State | MDS | Activity | dns | inos |
>>>> +------+--------+----------------+---------------+-------+-------+
>>>> | 0 | active | stmailmds01d-3 | Reqs: 89 /s | 375k | 371k |
>>>> | 1 | active | stmailmds01d-4 | Reqs: 64 /s | 386k | 383k |
>>>> | 2 | active | stmailmds01a-3 | Reqs: 9 /s | 403k | 399k |
>>>> | 3 | active | stmailmds01a-8 | Reqs: 23 /s | 393k | 390k |
>>>> | 4 | active | stmailmds01a-2 | Reqs: 36 /s | 391k | 387k |
>>>> | 5 | active | stmailmds01a-4 | Reqs: 57 /s | 394k | 390k |
>>>> | 6 | active | stmailmds01a-6 | Reqs: 50 /s | 395k | 391k |
>>>> | 7 | active | stmailmds01d-5 | Reqs: 37 /s | 384k | 380k |
>>>> | 8 | active | stmailmds01a-5 | Reqs: 39 /s | 397k | 394k |
>>>> | 9 | active | stmailmds01a | Reqs: 23 /s | 400k | 396k |
>>>> | 10 | active | stmailmds01d-8 | Reqs: 74 /s | 402k | 399k |
>>>> | 11 | active | stmailmds01d-6 | Reqs: 37 /s | 399k | 395k |
>>>> | 12 | active | stmailmds01d | Reqs: 36 /s | 394k | 390k |
>>>> | 13 | active | stmailmds01d-7 | Reqs: 80 /s | 397k | 393k |
>>>> | 14 | active | stmailmds01d-2 | Reqs: 56 /s | 414k | 410k |
>>>> | 15 | active | stmailmds01a-7 | Reqs: 25 /s | 390k | 387k |
>>>> +------+--------+----------------+---------------+-------+-------+
>>>> +-----------------+----------+-------+-------+
>>>> | Pool | type | used | avail |
>>>> +-----------------+----------+-------+-------+
>>>> | cephfs_metadata | metadata | 25.4G | 16.1T |
>>>> | cephfs_data | data | 2078G | 16.1T |
>>>> +-----------------+----------+-------+-------+
>>>> +----------------+
>>>> | Standby MDS |
>>>> +----------------+
>>>> | stmailmds01b-5 |
>>>> | stmailmds01b-2 |
>>>> | stmailmds01b-3 |
>>>> | stmailmds01b |
>>>> | stmailmds01b-7 |
>>>> | stmailmds01b-8 |
>>>> | stmailmds01b-6 |
>>>> | stmailmds01b-4 |
>>>> +----------------+
>>>> MDS version: ceph version 14.2.22-404-gf74e15c2e55
>>>> (f74e15c2e552b3359f5a51482dfd8b049e262743) nautilus (stable)
>>>> ---snip---
>>>
>>>
>>>
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users@xxxxxxx
>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
