Hi, this is your Linux kernel regression tracker. CCing the regression mailing list, as it should be in the loop for all regressions, as explained here: https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html On 21.03.22 05:53, Bruno Damasceno Freire wrote: > Hello everybody; > > This regression was first found during rpm operations with specific packages that became A LOT slower to update ranging from 4 to 30 minutes. > > The slowness results from: > a_ the kernel regression: specific system calls touching files with btrfs compression property will generate higher inode eviction on 5.15 kernels. > b_ the inode eviction generating btrfs inode logging and directory logging. > c_ the btrfs directory logging on the 5.15 kernel not being particulary efficient in the presence of high inode eviction. > > There is already an ongoing work [1] to improve "c" on newer kernels but I was told they are not elegible for the 5.15 version due to backporting policy restrictions. > AFAIK there isn't any work for "a" yet. > The consequence is that btrfs users running the 5.15 LTS kernel may experience severely degraded performance for specific I/O workloads on files with the compression property enabled. > > ___How to reproduce: > After some research I learned how to reproduce the regression without rpm. > > 1st option) > I made a script specifically to research this regression [2]. > It has more information, more test results and several options. > The scrip does a little too much so I'm just linking it here. > I hope it can help. > > 2nd option) > boot a 5.15 kernel, > setup and mount a RAM disk with btrfs, > create a folder and set its compression property, > populate the folder, > make a loop that: > -rename a file, > -unlink the renamed file, > -create a new file. > > [1] https://bugzilla.opensuse.org/show_bug.cgi?id=1193549 > [2] https://github.com/bdamascen0/s3e > > ___Test results > These tests were done on a virtual machine (kvm) with Ubuntu Jammy Jellyfish. > The kernel is 5.15.0.23 that relates to the 5.15.27 upstream kernel. Please repeat this with a vanilla kernel, Ubuntu's kernel are heavily patched and one of their patches might be causing your problem. > Main results (x86_64): > 250 files - zstd: 17521 ms @inode_evictions: 31375 > 250 files - lzo: 17114 ms @inode_evictions: 31375 > 250 files - uncompressed: 1138 ms @inode_evictions: 499 > > Load test results (x86_64): > 1000 files - 51.6 x more inode evictions - 18.1 x more time > 250 files - 62.9 x more inode evictions - 15.2 x more time > 100 files - 25.4 x more inode evictions - 3.7 x more time > 50 files - 12.8 x more inode evictions - 2.0 x more time > 10 files - 2.8 x more inode evictions - 1.3 x more time I'm missing something: more inode evictions when compared to what? A 5.14 vanilla kernel? > CPU usage results (x86_64): > 1000 files - zstd: 137841 ms > real 2m17,881s > user 0m1,704s > sys 2m11,937s > 1000 files - lzo: 135456 ms > real 2m15,478s > user 0m1,805s > sys 2m9,758s > 1000 files - uncompressed: 7496 ms > real 0m7,517s > user 0m1,386s > sys 0m4,899s > > I'm sending this message to the linux-fsdevel mailing list first. > Please tell if you think this subject would be of interest of another kernel subsystem. > PS: I'm not subscribed to this list. We need to get Btrfs people into the boat, but please clarify first if this really is a regression with the upstream kernel. Ciao, Thorsten
