Hi,
We sometimes have similar stuck client recall warnings.
To debug you can try:
(1) ceph health detail
that will show you the client ids which are generating the
warning. (e.g. 1234)
(2) ceph tell mds.* client ls id=1234
this will show lots of client statistics for the session. Notably
you need to look at the num_caps and the various recall metrics, and
see how they are changing over time.
>From experience this can be one of two or three things:
1. the client is very quickly iterating through files and the mds
isn't recalling them fast enough -- that shouldn't be the case if
you're running the default caps recall tuning on 16.2.6.
2. the client has some files open and cannot release the caps it holds
for some reason. We've seen some apps behave like this, and also I've
noticed that if users mount cephfs on another cephfs, or use bind
mounts, in some cases the caps of the "lower" cephfs will simply never
be released. I've found that issuing an `ls -lR` in the working dir of
the other client sometimes provokes the caps to be released. Or, if
they really will never release, you can try `echo 2 >
/proc/sys/vm/drop_caches` or umount / mount on the client.
If the cause isn't obvious to you, you can share some client session
stats and we can try to help here.
Best Regards,
dan
On Thu, Nov 18, 2021 at 6:36 AM 胡 玮文 <huww98@xxxxxxxxxxx> wrote:
>
> Hi all,
>
> We are consistently seeing the MDS_CLIENT_RECALL warning in our cluster, it seems harmless, but we cannot get HEALTH_OK, which is annoying.
>
> The clients that are reported failing to respond to cache pressure are constantly changing, and most of the time we got 1-5 such clients out of ~20. All of the clients are kernel clients, running HWE kernel 5.11 of Ubuntu 20.04. The load is pretty low.
>
> We are reading datasets that consist of millions of small files from cephfs, so we have tuned some config for performance. Some configs from "ceph config dump" that might be relevant:
>
> WHO LEVEL OPTION VALUE
> mds basic mds_cache_memory_limit 51539607552
> mds advanced mds_max_caps_per_client 8388608
> client basic client_cache_size 32768
>
> We also manually pinned almost every directory to either rank 0 or rank 1.
>
> Any thoughts about what causes the warning, or how can we get rid of it?
>
> Thanks,
> Weiwen Hu
>
>
> # ceph -s
> cluster:
> id: e88d509a-f6fc-11ea-b25d-a0423f3ac864
> health: HEALTH_WARN
> 4 clients failing to respond to cache pressure
>
> services:
> mon: 5 daemons, quorum gpu024,gpu006,gpu023,gpu013,gpu018 (age 7d)
> mgr: gpu014.kwbqcf(active, since 2w), standbys: gpu024.bapbcz
> mds: 2/2 daemons up, 2 hot standby
> osd: 45 osds: 45 up (since 2h), 45 in (since 5d)
> rgw: 2 daemons active (2 hosts, 1 zones)
>
> data:
> volumes: 1/1 healthy
> pools: 16 pools, 1713 pgs
> objects: 265.84M objects, 55 TiB
> usage: 115 TiB used, 93 TiB / 208 TiB avail
> pgs: 1711 active+clean
> 2 active+clean+scrubbing+deep
>
> io:
> client: 55 MiB/s rd, 5.2 MiB/s wr, 513 op/s rd, 14 op/s wr
>
>
> # ceph fs status
> cephfs - 23 clients
> ======
> RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
> 0 active cephfs.gpu018.ovxvoz Reqs: 241 /s 17.3M 17.3M 41.3k 5054k
> 1 active cephfs.gpu023.aetiph Reqs: 1 /s 13.1M 12.1M 864k 586k
> 1-s standby-replay cephfs.gpu006.ddpekw Evts: 2 /s 2517k 2393k 216k 0
> 0-s standby-replay cephfs.gpu024.rpfbnh Evts: 17 /s 9587k 9587k 214k 0
> POOL TYPE USED AVAIL
> cephfs.cephfs.meta metadata 126G 350G
> cephfs.cephfs.data data 102T 25.9T
> cephfs.cephfs.data_ssd data 0 525G
> cephfs.cephfs.data_mixed data 9.81T 350G
> cephfs.cephfs.data_ec data 166G 41.4T
> MDS version: ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable)
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
