MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to capability release

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,
I'm using cephfs as a storage layer for a database.
And seeing the following message in the health warning of ceph-
```
# ceph health detail
HEALTH_WARN 1 clients failing to respond to capability release
[WRN] MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to capability
release
    mds.ceph-mon-2(mds.0): Client client-3-4:cephfs failing to respond to
capability release client_id: 2100909
```
And the `mds session ls` output for the client is:
```
    {
        "id": 2100909,
        "entity": {
            "name": {
                "type": "client",
                "num": 2100909
            },
            "addr": {
                "type": "v1",
                "addr": "xxxx",
                "nonce": 2770840461
            }
        },
        "state": "open",
        "num_leases": 0,
        "num_caps": 507799,
        "request_load_avg": 1066,
        "uptime": 274389.13036294398,
        "requests_in_flight": 0,
        "completed_requests": 0,
        "reconnecting": false,
        "recall_caps": {
            "value": 0,
            "halflife": 60
        },
        "release_caps": {
            "value": 0,
            "halflife": 60
        },
        "recall_caps_throttle": {
            "value": 0,
            "halflife": 1.5
        },
        "recall_caps_throttle2o": {
            "value": 0,
            "halflife": 0.5
        },
        "session_cache_liveness": {
            "value": 9665.2693316477944,
            "halflife": 300
        },
        "cap_acquisition": {
            "value": 0,
            "halflife": 10
        },
        "inst": "client.2100909 v1:xxxx:0/2770840461",
        "completed_requests": [],
        "prealloc_inos": [],
        "used_inos": [],
        "client_metadata": {
            "client_features": {
                "feature_bits": "0x0000000000007bff"
            },
            "metric_spec": {
                "metric_flags": {
                    "feature_bits": "0x000000000000001f"
                }
            },
            "entity_id": "cephfs",
            "hostname": "client-3-4",
            "kernel_version": "5.11.0-1024-gcp",
            "root": "/"
        }
    },
```
ceph version: `15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus
(stable)`
client kernel version: `Linux 5.11.0-1023-gcp`
debug details-
```
# cat /sys/kernel/debug/ceph/*/osdc
REQUESTS 0 homeless 0
LINGER REQUESTS
BACKOFFS

# cat /sys/kernel/debug/ceph/*/caps
total           508281
avail           45
used            508230
reserved        6
min             1024

# Nothing in mdsc
```
The above warning seems to go away after some time, but it pops up multiple
times a day with different clients. It also seems to go away for some time
after I drop the cache using `echo 3 > /proc/sys/vm/drop_caches`.
Please suggest how do I resolve this issue permanently?

Best,
Anmol Arora
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux