Hi all, I've been investigating why system (kernel-mode) CPU usage of our hosts keeps climbing up over time (over several weeks). I realized it can be remedied by dropping cache w: $ echo 2 > /proc/sys/vm/drop_caches which significantly reduces the system CPU usage like below w/o any other change: https://photos.app.goo.gl/aYQmi5v6qnThbt8S7 I was able to repro the same by running: $ stress-ng -o 10 to generate open() sys calls, and later by opening non-existent files to increase negative dentry count to ~8.6 million: $ cat /proc/sys/fs/dentry-state 11039972 10987061 45 0 8615033 0 Before stress-ng was consuming 20% system CPU (negative dentry count was around 2 million) and after it started to consume +40% (w/ 8.6 million). https://photos.app.goo.gl/xc3m5fwZdbqgbdaM7 I tried the same on another host by increasing regular dentry count to ~80 million by iterating over existing files. There was no regression. Why would negative dentries have such a profound impact on open() sys call performance? Linux version: $ uname -r 5.14.0-284.11.1.el9_2.x86_64 Thanks! P.S. I've perf top / perf record / flamegraphs if needed.
