Hello Eugen,
did you have some success with modifying the mentioned values?
Or some others from:
https://docs.ceph.com/en/latest/cephfs/cache-configuration/
Best,
Malte
Am 15.06.22 um 14:12 schrieb Eugen Block:
Hi *,
I finally caught some debug logs during the cache pressure warnings. In
the meantime I had doubled the mds_cache_memory_limit to 128 GB which
decreased the number cache pressure messages significantly, but they
still appear a few times per day.
Turning on debug logs for a few seconds results in a 1 GB file, but I
found this message:
2022-06-15 10:07:34.254 7fdbbd44a700 2 mds.beacon.stmailmds01b-8
Session chead015:cephfs_client (2757628057) is not releasing caps fast
enough. Recalled caps at 390118 > 262144 (mds_recall_warning_threshold).
So now I know which limit is reached here, the question is what to do
about it? Should I increase the mds_recall_warning_threshold (default
256k) or should I maybe increase mds_recall_max_caps (currently at 60k,
default is 50k)? Any other suggestions? I'd appreciate any comments.
Thanks,
Eugen
Zitat von Eugen Block <eblock@xxxxxx>:
Hi,
I'm currently debugging a reoccuring issue with multi-active MDS. The
cluster is still on Nautilus and can't be upgraded at this time. There
have been many discussions about "cache pressure" and I was able to
find the right settings a couple of times, but before I change too
much in this setup I'd like to ask for your opinion. I'll add some
information at the end.
So we have 16 active MDS daemons spread over 2 servers for one cephfs
(8 daemons per server) with mds_cache_memory_limit = 64GB, the MDS
servers are mostly idle except for some short peaks. Each of the MDS
daemons uses around 2 GB according to 'ceph daemon mds.<MDS> cache
status', so we're nowhere near the 64GB limit. There are currently 25
servers that mount the cephs as clients.
Watching the ceph health I can see that the reported clients with
cache pressure change, so they are not actually stuck but just don't
respond as quickly as the MDS would like them to (I assume). For some
of the mentioned clients I see high values for .recall_caps.value in
the 'daemon session ls' output (at the bottom).
The docs basically state this:
When the MDS needs to shrink its cache (to stay within
mds_cache_size), it sends messages to clients to shrink their caches
too. The client is unresponsive to MDS requests to release cached
inodes. Either the client is unresponsive or has a bug
To me it doesn't seem like the MDS servers are near the cache size
limit, so it has to be the clients, right? In a different setup it
helped to decrease the client_oc_size from 200MB to 100MB, but then
there's also client_cache_size with 16K default. I'm not sure what the
best approach would be here. I'd appreciate any comments on how to
size the various cache/caps/threshold configurations.
Thanks!
Eugen
---snip---
# ceph daemon mds.<MDS> session ls
"id": 2728101146,
"entity": {
"name": {
"type": "client",
"num": 2728101146
},
[...]
"nonce": 1105499797
}
},
"state": "open",
"num_leases": 0,
"num_caps": 16158,
"request_load_avg": 0,
"uptime": 1118066.210318422,
"requests_in_flight": 0,
"completed_requests": [],
"reconnecting": false,
"recall_caps": {
"value": 788916.8276369586,
"halflife": 60
},
"release_caps": {
"value": 8.814981576458962,
"halflife": 60
},
"recall_caps_throttle": {
"value": 27379.27162576508,
"halflife": 1.5
},
"recall_caps_throttle2o": {
"value": 5382.261925615086,
"halflife": 0.5
},
"session_cache_liveness": {
"value": 12.91841737465921,
"halflife": 300
},
"cap_acquisition": {
"value": 0,
"halflife": 10
},
[...]
"used_inos": [],
"client_metadata": {
"features": "0x0000000000003bff",
"entity_id": "cephfs_client",
# ceph fs status
cephfs - 25 clients
======
+------+--------+----------------+---------------+-------+-------+
| Rank | State | MDS | Activity | dns | inos |
+------+--------+----------------+---------------+-------+-------+
| 0 | active | stmailmds01d-3 | Reqs: 89 /s | 375k | 371k |
| 1 | active | stmailmds01d-4 | Reqs: 64 /s | 386k | 383k |
| 2 | active | stmailmds01a-3 | Reqs: 9 /s | 403k | 399k |
| 3 | active | stmailmds01a-8 | Reqs: 23 /s | 393k | 390k |
| 4 | active | stmailmds01a-2 | Reqs: 36 /s | 391k | 387k |
| 5 | active | stmailmds01a-4 | Reqs: 57 /s | 394k | 390k |
| 6 | active | stmailmds01a-6 | Reqs: 50 /s | 395k | 391k |
| 7 | active | stmailmds01d-5 | Reqs: 37 /s | 384k | 380k |
| 8 | active | stmailmds01a-5 | Reqs: 39 /s | 397k | 394k |
| 9 | active | stmailmds01a | Reqs: 23 /s | 400k | 396k |
| 10 | active | stmailmds01d-8 | Reqs: 74 /s | 402k | 399k |
| 11 | active | stmailmds01d-6 | Reqs: 37 /s | 399k | 395k |
| 12 | active | stmailmds01d | Reqs: 36 /s | 394k | 390k |
| 13 | active | stmailmds01d-7 | Reqs: 80 /s | 397k | 393k |
| 14 | active | stmailmds01d-2 | Reqs: 56 /s | 414k | 410k |
| 15 | active | stmailmds01a-7 | Reqs: 25 /s | 390k | 387k |
+------+--------+----------------+---------------+-------+-------+
+-----------------+----------+-------+-------+
| Pool | type | used | avail |
+-----------------+----------+-------+-------+
| cephfs_metadata | metadata | 25.4G | 16.1T |
| cephfs_data | data | 2078G | 16.1T |
+-----------------+----------+-------+-------+
+----------------+
| Standby MDS |
+----------------+
| stmailmds01b-5 |
| stmailmds01b-2 |
| stmailmds01b-3 |
| stmailmds01b |
| stmailmds01b-7 |
| stmailmds01b-8 |
| stmailmds01b-6 |
| stmailmds01b-4 |
+----------------+
MDS version: ceph version 14.2.22-404-gf74e15c2e55
(f74e15c2e552b3359f5a51482dfd8b049e262743) nautilus (stable)
---snip---
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
