Re: [EXTERN] cache pressure?

Dietmar Rieder <dietmar.rieder@xxxxxxxxxxx> · Wed, 24 Apr 2024 21:58:50 +0200

Hi Erich,

in our case the "client failing to respond to cache pressure" situation 
is/was often caused by users how have vscode connecting via ssh to our 
HPC head node. vscode makes heavy use of file watchers and we have seen 
users with > 400k watchers. All these watched files must be held in the 
MDS cache and if you have multiple users at the same time running vscode 
it gets problematic.

Unfortunately there is no global setting - at least none that we are 
aware of - for vscode to exclude certain files or directories from being 
watched. We asked the users to configure their vscode (Remote Settings 
-> Watcher Exclude) as follows:

{
  "files.watcherExclude": {
     "**/.git/objects/**": true,
     "**/.git/subtree-cache/**": true,
     "**/node_modules/*/**": true,
    "**/.cache/**": true,
    "**/.conda/**": true,
    "**/.local/**": true,
    "**/.nextflow/**": true,
    "**/work/**": true
  }
}

~/.vscode-server/data/Machine/settings.json

To monitor and find processes with watcher you may use inotify-info
<https://github.com/mikesart/inotify-info>

HTH
  Dietmar

On 4/23/24 15:47, Erich Weiler wrote:
So I'm trying to figure out ways to reduce the number of warnings I'm 
getting and I'm thinking about the one "client failing to respond to 
cache pressure".

Is there maybe a way to tell a client (or all clients) to reduce the 
amount of cache it uses or to release caches quickly?  Like, all the time?

I know the linux kernel (and maybe ceph) likes to cache everything for a 
while, and rightfully so, but I suspect in my use case it may be more 
efficient to more quickly purge the cache or to in general just cache 
way less overall...?

We have many thousands of threads all doing different things that are 
hitting our filesystem, so I suspect the caching isn't really doing me 
much good anyway due to the churn, and probably is causing more problems 
than it helping...

-erich
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

Attachment:
OpenPGP_signature.asc

Description: OpenPGP digital signature
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx