Hello everybody; This regression was first found during rpm operations with specific packages that became A LOT slower to update ranging from 4 to 30 minutes. The slowness results from: a_ the kernel regression: specific system calls touching files with btrfs compression property will generate higher inode eviction on 5.15 kernels. b_ the inode eviction generating btrfs inode logging and directory logging. c_ the btrfs directory logging on the 5.15 kernel not being particulary efficient in the presence of high inode eviction. There is already an ongoing work [1] to improve "c" on newer kernels but I was told they are not elegible for the 5.15 version due to backporting policy restrictions. AFAIK there isn't any work for "a" yet. The consequence is that btrfs users running the 5.15 LTS kernel may experience severely degraded performance for specific I/O workloads on files with the compression property enabled. ___How to reproduce: After some research I learned how to reproduce the regression without rpm. 1st option) I made a script specifically to research this regression [2]. It has more information, more test results and several options. The scrip does a little too much so I'm just linking it here. I hope it can help. 2nd option) boot a 5.15 kernel, setup and mount a RAM disk with btrfs, create a folder and set its compression property, populate the folder, make a loop that: -rename a file, -unlink the renamed file, -create a new file. [1] https://bugzilla.opensuse.org/show_bug.cgi?id=1193549 [2] https://github.com/bdamascen0/s3e ___Test results These tests were done on a virtual machine (kvm) with Ubuntu Jammy Jellyfish. The kernel is 5.15.0.23 that relates to the 5.15.27 upstream kernel. Main results (x86_64): 250 files - zstd: 17521 ms @inode_evictions: 31375 250 files - lzo: 17114 ms @inode_evictions: 31375 250 files - uncompressed: 1138 ms @inode_evictions: 499 Load test results (x86_64): 1000 files - 51.6 x more inode evictions - 18.1 x more time 250 files - 62.9 x more inode evictions - 15.2 x more time 100 files - 25.4 x more inode evictions - 3.7 x more time 50 files - 12.8 x more inode evictions - 2.0 x more time 10 files - 2.8 x more inode evictions - 1.3 x more time CPU usage results (x86_64): 1000 files - zstd: 137841 ms real 2m17,881s user 0m1,704s sys 2m11,937s 1000 files - lzo: 135456 ms real 2m15,478s user 0m1,805s sys 2m9,758s 1000 files - uncompressed: 7496 ms real 0m7,517s user 0m1,386s sys 0m4,899s I'm sending this message to the linux-fsdevel mailing list first. Please tell if you think this subject would be of interest of another kernel subsystem. PS: I'm not subscribed to this list. TIA, Bruno
