Hi Burkhard, This is a known issue. We ran into it a few months back using VScode containers working on Cephfs under Kubernetes. Tweaking the settings.json file as he suggested here [1] by Dietmar did the trick for us. Regards, Frédéric. [1] https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/message/EUCRAYM3JEGK6KCJND7SZNB3HYHJAB65/ ----- Le 25 Sep 24, à 9:55, Burkhard Linke Burkhard.Linke@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx a écrit : > Hi, > > > we are currently trying to debug and understand a problem with cephfs > and inotify watchers. A user is running Visual Studio Code with a > workspace on a cephfs mount. VSC uses inotify for monitoring files and > directories in the workspace: > > > root@cli:~# ./inotify-info > ------------------------------------------------------------------------------ > INotify Limits: > max_queued_events 16,384 > max_user_instances 128 > max_user_watches 1,048,576 > ------------------------------------------------------------------------------ > Pid Uid App Watches Instances > 3599940 1236 node 1,681 1 > 1 0 systemd 106 5 > 3600170 1236 node 54 1 > 874797 0 udevadm 17 1 > 3599118 0 systemd 7 3 > 3599707 1236 systemd 7 3 > 3599918 1236 node 6 1 > 2047 100 dbus-daemon 3 1 > 2054 0 sssd 2 1 > 2139 0 systemd-logind (deleted) 1 1 > 2446 0 agetty 1 1 > 3600001 1236 node 1 1 > ------------------------------------------------------------------------------ > Total inotify Watches: 1886 > Total inotify Instances: 20 > ------------------------------------------------------------------------------ > root@cli:~# cat /sys/kernel/debug/ceph/XYZ.client354064780/caps | wc -l > 1773083 > > root@cli:~# uname -a > Linux cli 6.1.0-23-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.99-1 > (2024-07-15) x86_64 GNU/Linux > > > So roughly 1.700 watchers result in over 1.7 million caps. (some of the > watchers might for files on different filesystems). I've also checked > this on the MDS side, it also reports a very high number of caps for > that client. Running tools like lsof on the host as root only reports > very few open files (<50). So inotify seems to be responsible for the > massive caps build up. Terminating VSC results in a sharp drop of the > caps (just a few open files / directories left afterwards). > > > Is this a known problem? > > > Best regards, > > Burkhard Linke > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |