On 24.03.22 13:18, Thorsten Leemhuis wrote: > Hi, this is your Linux kernel regression tracker. > > CCing the regression mailing list, as it should be in the loop for all > regressions, as explained here: > https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html Thanks. > On 21.03.22 05:53, Bruno Damasceno Freire wrote: > > Hello everybody; > > > > This regression was first found during rpm operations with specific packages that became A LOT slower to update ranging from 4 to 30 minutes. > > > > The slowness results from: > > a_ the kernel regression: specific system calls touching files with btrfs compression property will generate higher inode eviction on 5.15 kernels. > > b_ the inode eviction generating btrfs inode logging and directory logging. > > c_ the btrfs directory logging on the 5.15 kernel not being particulary efficient in the presence of high inode eviction. > > > > There is already an ongoing work [1] to improve "c" on newer kernels but I was told they are not elegible for the 5.15 version due to backporting policy restrictions. > > AFAIK there isn't any work for "a" yet. > > The consequence is that btrfs users running the 5.15 LTS kernel may experience severely degraded performance for specific I/O workloads on files with the compression property enabled. > > > > ___How to reproduce: > > After some research I learned how to reproduce the regression without rpm. > > > > 1st option) > > I made a script specifically to research this regression [2]. > > It has more information, more test results and several options. > > The scrip does a little too much so I'm just linking it here. > > I hope it can help. > > > > 2nd option) > > boot a 5.15 kernel, > > setup and mount a RAM disk with btrfs, > > create a folder and set its compression property, > > populate the folder, > > make a loop that: > > -rename a file, > > -unlink the renamed file, > > -create a new file. > > > > [1] https://bugzilla.opensuse.org/show_bug.cgi?id=1193549 > > [2] https://github.com/bdamascen0/s3e > > > > ___Test results > > These tests were done on a virtual machine (kvm) with Ubuntu Jammy Jellyfish. > > The kernel is 5.15.0.23 that relates to the 5.15.27 upstream kernel. > > Please repeat this with a vanilla kernel, Ubuntu's kernel are heavily > patched and one of their patches might be causing your problem. Done. Same higher inode-evictions and slow results. Also tested on a 5.15.0-rc1 kernel and, again, same results. Older and newer 5.14 kernels produced normal results so it really seems that the regression was introduced early in the 5.15 cycle. > > Main results (x86_64): > > 250 files - zstd: 17521 ms @inode_evictions: 31375 > > 250 files - lzo: 17114 ms @inode_evictions: 31375 > > 250 files - uncompressed: 1138 ms @inode_evictions: 499 > > > > Load test results (x86_64): > > 1000 files - 51.6 x more inode evictions - 18.1 x more time > > 250 files - 62.9 x more inode evictions - 15.2 x more time > > 100 files - 25.4 x more inode evictions - 3.7 x more time > > 50 files - 12.8 x more inode evictions - 2.0 x more time > > 10 files - 2.8 x more inode evictions - 1.3 x more time > > I'm missing something: more inode evictions when compared to what? A > 5.14 vanilla kernel? Not clear at all. Sorry. Will repost with my full set of results [2] including a vanilla kernel, the 15.0-rc1, more distros and a better load test report. > > CPU usage results (x86_64): > > 1000 files - zstd: 137841 ms > > real 2m17,881s > > user 0m1,704s > > sys 2m11,937s > > 1000 files - lzo: 135456 ms > > real 2m15,478s > > user 0m1,805s > > sys 2m9,758s > > 1000 files - uncompressed: 7496 ms > > real 0m7,517s > > user 0m1,386s > > sys 0m4,899s > > > > I'm sending this message to the linux-fsdevel mailing list first. > > Please tell if you think this subject would be of interest of another kernel subsystem. > > PS: I'm not subscribed to this list. > > We need to get Btrfs people into the boat, but please clarify first if > this really is a regression with the upstream kernel. Sure. The Btrfs crew belong on this boat. The Btrfs developer Filipe Manana assisted me on the original bug report [1]. He concluded that the btrfs compression property just happens to trigger something outside the btrfs code and was guessing it could be at the vfs or mm subsystems. The bug was also the beginning of a long term work to make the filesystem more robust but the improvements would not be elegible for the 5.15 version due to backport restrictions. > Ciao, Thorsten Grazie, Bruno
