Re: Multi-active MDS cache pressure

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi *,

I finally caught some debug logs during the cache pressure warnings. In the meantime I had doubled the mds_cache_memory_limit to 128 GB which decreased the number cache pressure messages significantly, but they still appear a few times per day.

Turning on debug logs for a few seconds results in a 1 GB file, but I found this message:

2022-06-15 10:07:34.254 7fdbbd44a700 2 mds.beacon.stmailmds01b-8 Session chead015:cephfs_client (2757628057) is not releasing caps fast enough. Recalled caps at 390118 > 262144 (mds_recall_warning_threshold).

So now I know which limit is reached here, the question is what to do about it? Should I increase the mds_recall_warning_threshold (default 256k) or should I maybe increase mds_recall_max_caps (currently at 60k, default is 50k)? Any other suggestions? I'd appreciate any comments.

Thanks,
Eugen


Zitat von Eugen Block <eblock@xxxxxx>:

Hi,

I'm currently debugging a reoccuring issue with multi-active MDS. The cluster is still on Nautilus and can't be upgraded at this time. There have been many discussions about "cache pressure" and I was able to find the right settings a couple of times, but before I change too much in this setup I'd like to ask for your opinion. I'll add some information at the end. So we have 16 active MDS daemons spread over 2 servers for one cephfs (8 daemons per server) with mds_cache_memory_limit = 64GB, the MDS servers are mostly idle except for some short peaks. Each of the MDS daemons uses around 2 GB according to 'ceph daemon mds.<MDS> cache status', so we're nowhere near the 64GB limit. There are currently 25 servers that mount the cephs as clients. Watching the ceph health I can see that the reported clients with cache pressure change, so they are not actually stuck but just don't respond as quickly as the MDS would like them to (I assume). For some of the mentioned clients I see high values for .recall_caps.value in the 'daemon session ls' output (at the bottom).

The docs basically state this:
When the MDS needs to shrink its cache (to stay within mds_cache_size), it sends messages to clients to shrink their caches too. The client is unresponsive to MDS requests to release cached inodes. Either the client is unresponsive or has a bug

To me it doesn't seem like the MDS servers are near the cache size limit, so it has to be the clients, right? In a different setup it helped to decrease the client_oc_size from 200MB to 100MB, but then there's also client_cache_size with 16K default. I'm not sure what the best approach would be here. I'd appreciate any comments on how to size the various cache/caps/threshold configurations.

Thanks!
Eugen


---snip---
# ceph daemon mds.<MDS> session ls

    "id": 2728101146,
    "entity": {
      "name": {
        "type": "client",
        "num": 2728101146
      },
[...]
        "nonce": 1105499797
      }
    },
    "state": "open",
    "num_leases": 0,
    "num_caps": 16158,
    "request_load_avg": 0,
    "uptime": 1118066.210318422,
    "requests_in_flight": 0,
    "completed_requests": [],
    "reconnecting": false,
    "recall_caps": {
      "value": 788916.8276369586,
      "halflife": 60
    },
    "release_caps": {
      "value": 8.814981576458962,
      "halflife": 60
    },
    "recall_caps_throttle": {
      "value": 27379.27162576508,
      "halflife": 1.5
    },
    "recall_caps_throttle2o": {
      "value": 5382.261925615086,
      "halflife": 0.5
    },
    "session_cache_liveness": {
      "value": 12.91841737465921,
      "halflife": 300
    },
    "cap_acquisition": {
      "value": 0,
      "halflife": 10
    },
[...]
    "used_inos": [],
    "client_metadata": {
      "features": "0x0000000000003bff",
      "entity_id": "cephfs_client",


# ceph fs status

cephfs - 25 clients
======
+------+--------+----------------+---------------+-------+-------+
| Rank | State  |      MDS       |    Activity   |  dns  |  inos |
+------+--------+----------------+---------------+-------+-------+
|  0   | active | stmailmds01d-3 | Reqs:   89 /s |  375k |  371k |
|  1   | active | stmailmds01d-4 | Reqs:   64 /s |  386k |  383k |
|  2   | active | stmailmds01a-3 | Reqs:    9 /s |  403k |  399k |
|  3   | active | stmailmds01a-8 | Reqs:   23 /s |  393k |  390k |
|  4   | active | stmailmds01a-2 | Reqs:   36 /s |  391k |  387k |
|  5   | active | stmailmds01a-4 | Reqs:   57 /s |  394k |  390k |
|  6   | active | stmailmds01a-6 | Reqs:   50 /s |  395k |  391k |
|  7   | active | stmailmds01d-5 | Reqs:   37 /s |  384k |  380k |
|  8   | active | stmailmds01a-5 | Reqs:   39 /s |  397k |  394k |
|  9   | active |  stmailmds01a  | Reqs:   23 /s |  400k |  396k |
|  10  | active | stmailmds01d-8 | Reqs:   74 /s |  402k |  399k |
|  11  | active | stmailmds01d-6 | Reqs:   37 /s |  399k |  395k |
|  12  | active |  stmailmds01d  | Reqs:   36 /s |  394k |  390k |
|  13  | active | stmailmds01d-7 | Reqs:   80 /s |  397k |  393k |
|  14  | active | stmailmds01d-2 | Reqs:   56 /s |  414k |  410k |
|  15  | active | stmailmds01a-7 | Reqs:   25 /s |  390k |  387k |
+------+--------+----------------+---------------+-------+-------+
+-----------------+----------+-------+-------+
|       Pool      |   type   |  used | avail |
+-----------------+----------+-------+-------+
| cephfs_metadata | metadata | 25.4G | 16.1T |
|   cephfs_data   |   data   | 2078G | 16.1T |
+-----------------+----------+-------+-------+
+----------------+
|  Standby MDS   |
+----------------+
| stmailmds01b-5 |
| stmailmds01b-2 |
| stmailmds01b-3 |
|  stmailmds01b  |
| stmailmds01b-7 |
| stmailmds01b-8 |
| stmailmds01b-6 |
| stmailmds01b-4 |
+----------------+
MDS version: ceph version 14.2.22-404-gf74e15c2e55 (f74e15c2e552b3359f5a51482dfd8b049e262743) nautilus (stable)
---snip---



_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux