Hi Btrfs developers, this is your Linux kernel regression tracker. Top-posting for once, to make this easily accessible to everyone. Are there any known regressions in 5.15 that cause more inode evictions? There is a user reporting such problems, see the msg quoted below, which is the last from this thread: https://lore.kernel.org/all/MN2PR20MB2512314446801B92562E26B5D2169@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/ @Bruno: sorry, you report is very hard to follow at least for me. Maybe the btrfs developers have a idea what's wrong here, but if not you likely need to make this easier for all us: Write a really simple testcase and run it on those vanilla kernels that matter, which are in this case: The latest 5.17.y and 5.15.y releases to check if it was already solved -- and if not, on 5.14 and 5.15. Maybe you need to bisect the issue to bring us closer the the root of the problem. But with a bit of luck the btrfs developers might have an idea already. Ciao, Thosten On 28.03.22 05:12, Bruno Damasceno Freire wrote: > ___Updated test results > > These results were obtained with the script I've made to research this regression [1]. > [1] https://github.com/bdamascen0/s3e > > > Main results > > The regression was reproduced on: > -several different 5.15 kernels versions across several different distros. > -all 5.15 kernels that I have tried on. > -the 5.15.0-rc1 kernel from the opensuse tumbleweed comunity repository. > -the 5.15.12 vanilla kernel from the official opensuse tumbleweed repository. > > The regression could not be reproduced on kernels versions other than the 5.15. > > The vanilla kernel test was suggested by Thorsten Leemhuis to make sure downstream custom patches aren't causing the symptoms. > The vanilla kernel test result shows the exact same pattern verified on downstream kernels and fully validates the regression. > > > General test results for the 5.15 kernel series (x86_64) > > opensuse tumbleweed ----- kernel 5.15.12 --- vanilla --- (kvm) > ... 250 files - zstd: 13327 ms @inode_evictions: 31375 > ... 250 files - lzo: 13361 ms @inode_evictions: 31375 > ... 250 files - uncompressed: 1204 ms @inode_evictions: 499 > opensuse tumbleweed ----- kernel 5.15.0-rc1-1.g8787773 - (kvm) > ... 250 files - zstd: 13875 ms @inode_evictions: 31375 > ... 250 files - lzo: 15351 ms @inode_evictions: 31375 > ... 250 files - uncompressed: 1231 ms @inode_evictions: 499 > opensuse tumbleweed ----- kernel 5.15.12---------------------- > ... 250 files - zstd: 12500 ms @inode_evictions: 31375 > ... 250 files - lzo: 12327 ms @inode_evictions: 31375 > ... 250 files - uncompressed: 1482 ms @inode_evictions: 499 > debian bookworm --------- kernel 5.15.0-3 - (5.15.15) -- (kvm) > ... 250 files - zstd: 12343 ms @inode_evictions: 31375 > ... 250 files - lzo: 14028 ms @inode_evictions: 31375 > ... 250 files - uncompressed: 1092 ms @inode_evictions: 499 > Zenwalk 15.0 Skywalker ---kernel 5.15.19 --------------- (kvm) > ... 250 files - zstd: 14374 ms @inode_evictions: - > ... 250 files - lzo: 14163 ms @inode_evictions: - > ... 250 files - uncompressed: 2173 ms @inode_evictions: - > ubuntu jammy jellyfish -- kernel 5.15.0.23 - (5.15.27) - (kvm) > ... 250 files - zstd: 17521 ms @inode_evictions: 31375 > ... 250 files - lzo: 17114 ms @inode_evictions: 31375 > ... 250 files - uncompressed: 1138 ms @inode_evictions: 499 > > > General test results for other kernels (x86_64) > > opensuse leap 15.3 ------ kernel 5.3.18-150300.59.54---------- > ... 250 files - zstd: 668 ms @inode_evictions: 251 > ... 250 files - lzo: 693 ms @inode_evictions: 251 > ... 250 files - uncompressed: 661 ms @inode_evictions: 252 > opensuse leap 15.4 beta - kernel 5.14.21-150400.11 ----- (kvm) > ... 250 files - zstd: 811 ms @inode_evictions: 251 > ... 250 files - lzo: 912 ms @inode_evictions: 251 > ... 250 files - uncompressed: 993 ms @inode_evictions: 251 > opensuse tumbleweed ----- kernel 5.14.14 --------------- (kvm) > ... 250 files - zstd: 888 ms @inode_evictions: 251 > ... 250 files - lzo: 1063 ms @inode_evictions: 251 > ... 250 files - uncompressed: 778 ms @inode_evictions: 251 > opensuse tumbleweed ----- kernel 5.16.14---------------------- > ... 250 files - zstd: 1398 ms @inode_evictions: 250 > ... 250 files - lzo: 1323 ms @inode_evictions: 250 > ... 250 files - uncompressed: 1365 ms @inode_evictions: 250 > > > Load test results (x86_64): > > opensuse leap 15.4 beta - kernel 5.14.21-150400.11 ----- (kvm) > ... 50 files - zstd: 261 ms @inode_evictions: 51 > ... 50 files - lzo: 256 ms @inode_evictions: 51 > ... 50 files - uncompressed: 317 ms @inode_evictions: 51 > ... 100 files - zstd: 450 ms @inode_evictions: 101 > ... 100 files - lzo: 461 ms @inode_evictions: 101 > ... 100 files - uncompressed: 471 ms @inode_evictions: 101 > ... 150 files - zstd: 618 ms @inode_evictions: 151 > ... 150 files - lzo: 624 ms @inode_evictions: 151 > ... 150 files - uncompressed: 612 ms @inode_evictions: 151 > ... 200 files - zstd: 822 ms @inode_evictions: 201 > ... 200 files - lzo: 933 ms @inode_evictions: 201 > ... 200 files - uncompressed: 747 ms @inode_evictions: 201 > ... 250 files - zstd: 1128 ms @inode_evictions: 251 > ... 250 files - lzo: 974 ms @inode_evictions: 251 > ... 250 files - uncompressed: 936 ms @inode_evictions: 251 > ... 1000 files - zstd: 3517 ms @inode_evictions: 1001 > ... 1000 files - lzo: 4373 ms @inode_evictions: 1001 > ... 1000 files - uncompressed: 3797 ms @inode_evictions: 1001 > ubuntu jammy jellyfish -- kernel 5.15.0.23 - (5.15.27) - (kvm) > ... 50 files - zstd: 424 ms @inode_evictions: 1275 > ... 50 files - lzo: 423 ms @inode_evictions: 1275 > ... 50 files - uncompressed: 207 ms @inode_evictions: 99 > ... 100 files - zstd: 1744 ms @inode_evictions: 5050 > ... 100 files - lzo: 1838 ms @inode_evictions: 5050 > ... 100 files - uncompressed: 373 ms @inode_evictions: 199 > ... 150 files - zstd: 4785 ms @inode_evictions: 11325 > ... 150 files - lzo: 4660 ms @inode_evictions: 11325 > ... 150 files - uncompressed: 689 ms @inode_evictions: 299 > ... 200 files - zstd: 9763 ms @inode_evictions: 20100 > ... 200 files - lzo: 10106 ms @inode_evictions: 20100 > ... 200 files - uncompressed: 938 ms @inode_evictions: 399 > ... 250 files - zstd: 17550 ms @inode_evictions: 31375 > ... 250 files - lzo: 17337 ms @inode_evictions: 31375 > ... 250 files - uncompressed: 1373 ms @inode_evictions: 499 > ... 1000 files - zstd: 143614 ms @inode_evictions: 101132 > ... 1000 files - lzo: 146724 ms @inode_evictions: 100314 > ... 1000 files - uncompressed: 7735 ms @inode_evictions: 1999 > > > Load test results comparisson for compressed files (x86_64): > ubuntu jammy jellyfish - compared to - opensuse leap 15.4 beta > > 50 files gives aprox. 1.6 x more time and aprox. 25 x more inode evictions > 100 files gives aprox. 3.8 x more time and aprox. 50 x more inode evictions > 150 files gives aprox. 7.4 x more time and aprox. 75 x more inode evictions > 200 files gives aprox. 10.8 x more time and aprox. 100 x more inode evictions > 250 files gives aprox. 15.5 x more time and aprox. 125 x more inode evictions > 1000 files gives aprox. 33.5 x more time and aprox. 100 x more inode evictions > > > Load test results comparisson for uncompressed files (x86_64): > ubuntu jammy jellyfish - compared to - opensuse leap 15.4 beta > > 50 files gives aprox. 0.6 x more time and aprox. 2 x more inode evictions > 100 files gives aprox. 0.8 x more time and aprox. 2 x more inode evictions > 150 files gives aprox. 1.1 x more time and aprox. 2 x more inode evictions > 200 files gives aprox. 1.2 x more time and aprox. 2 x more inode evictions > 250 files gives aprox. 1.4 x more time and aprox. 2 x more inode evictions > 1000 files gives aprox. 2.0 x more time and aprox. 2 x more inode evictions > > > CPU usage results: > The regression causes significant CPU usage by the kernel. > > ubuntu jammy jellyfish -- kernel 5.15.0.23 (5.15.27) - (kvm) > ... 1000 files - zstd: 137841 ms > real 2m17,881s > user 0m 1,704s > sys 2m11,937s > ... 1000 files - lzo: 135456 ms > real 2m15,478s > user 0m 1,805s > sys 2m 9,758s > ... 1000 files - uncompressed: 7496 ms > real 0m 7,517s > user 0m 1,386s > sys 0m 4,899s > > > Test system specification: > host: AMD FX-8370E 8 cores / 8GB RAM / ssd > guests (kvm): 2 cores / 2G RAM / ssd > test storage medium: RAM disk block device (host and guest) > > > TIA, Bruno
