Hi all, I have a question regarding the xfs slab memory usage when operating a filesystem with 1-2 billion inodes (raid 0 with 6 disks, totally 18TB). On this partition, whenever there is a high disk io operation, like removing millions of small files, the slab kernel memory usage will increase a lot, leading to many OOM issues happening for the services running on this node. You could check some of the stats as the following (only includes the xfs related): ######################################################################### Active / Total Objects (% used): 281803052 / 317485764 (88.8%) Active / Total Slabs (% used): 13033144 / 13033144 (100.0%) Active / Total Caches (% used): 126 / 180 (70.0%) Active / Total Size (% used): 114671057.99K / 127265108.19K (90.1%) Minium / Average / Maximum Object : 0.01K / 0.40K / 16.75K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 78207920 70947541 0% 1.00K 7731010 32 247392320K xfs_inode 59945928 46548798 0% 0.19K 1433102 42 11464816K dentry 25051296 25051282 0% 0.38K 599680 42 9594880K xfs_buf ######################################################################### The peak slab memory usage could spike all the way to 100GB+. We are using Ubuntu 18.04 and the xfs version is 4.9, kernel version is 5.4 ######################################################################### Linux# cat /etc/*-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS" NAME="Ubuntu" VERSION="18.04.5 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.5 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/"; SUPPORT_URL="https://help.ubuntu.com/"; BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"; PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"; VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic Linux# sudo apt list | grep xfs libguestfs-xfs/bionic-updates 1:1.36.13-1ubuntu3.3 amd64 nfs-ganesha-xfs/bionic 2.6.0-2 amd64 obexfs/bionic 0.11-2build1 amd64 x11-xfs-utils/bionic 7.7+2build1 amd64 xfsdump/bionic 3.1.6+nmu2 amd64 xfslibs-dev/bionic 4.9.0+nmu1ubuntu2 amd64 xfsprogs/bionic,now 4.9.0+nmu1ubuntu2 amd64 [installed] xfstt/bionic 1.9.3-3 amd64 xfswitch-plugin/bionic 0.0.1-5ubuntu5 amd64 Linux# uname -a Linux linux-host 5.4.0-45-generic #49~18.04.2-Ubuntu SMP Wed Aug 26 16:29:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux ######################################################################### Is there any potential way to limit the slab memory increase for a node as a whole, or the only thing we could do is to reduce the filesystem inode or iops usage? Thanks in advance! Jianan Wang
