___Updated test results These results were obtained with the script I've made to research this regression [1]. [1] https://github.com/bdamascen0/s3e Main results The regression was reproduced on: -several different 5.15 kernels versions across several different distros. -all 5.15 kernels that I have tried on. -the 5.15.0-rc1 kernel from the opensuse tumbleweed comunity repository. -the 5.15.12 vanilla kernel from the official opensuse tumbleweed repository. The regression could not be reproduced on kernels versions other than the 5.15. The vanilla kernel test was suggested by Thorsten Leemhuis to make sure downstream custom patches aren't causing the symptoms. The vanilla kernel test result shows the exact same pattern verified on downstream kernels and fully validates the regression. General test results for the 5.15 kernel series (x86_64) opensuse tumbleweed ----- kernel 5.15.12 --- vanilla --- (kvm) .. 250 files - zstd: 13327 ms @inode_evictions: 31375 .. 250 files - lzo: 13361 ms @inode_evictions: 31375 .. 250 files - uncompressed: 1204 ms @inode_evictions: 499 opensuse tumbleweed ----- kernel 5.15.0-rc1-1.g8787773 - (kvm) .. 250 files - zstd: 13875 ms @inode_evictions: 31375 .. 250 files - lzo: 15351 ms @inode_evictions: 31375 .. 250 files - uncompressed: 1231 ms @inode_evictions: 499 opensuse tumbleweed ----- kernel 5.15.12---------------------- .. 250 files - zstd: 12500 ms @inode_evictions: 31375 .. 250 files - lzo: 12327 ms @inode_evictions: 31375 .. 250 files - uncompressed: 1482 ms @inode_evictions: 499 debian bookworm --------- kernel 5.15.0-3 - (5.15.15) -- (kvm) .. 250 files - zstd: 12343 ms @inode_evictions: 31375 .. 250 files - lzo: 14028 ms @inode_evictions: 31375 .. 250 files - uncompressed: 1092 ms @inode_evictions: 499 Zenwalk 15.0 Skywalker ---kernel 5.15.19 --------------- (kvm) .. 250 files - zstd: 14374 ms @inode_evictions: - .. 250 files - lzo: 14163 ms @inode_evictions: - .. 250 files - uncompressed: 2173 ms @inode_evictions: - ubuntu jammy jellyfish -- kernel 5.15.0.23 - (5.15.27) - (kvm) .. 250 files - zstd: 17521 ms @inode_evictions: 31375 .. 250 files - lzo: 17114 ms @inode_evictions: 31375 .. 250 files - uncompressed: 1138 ms @inode_evictions: 499 General test results for other kernels (x86_64) opensuse leap 15.3 ------ kernel 5.3.18-150300.59.54---------- .. 250 files - zstd: 668 ms @inode_evictions: 251 .. 250 files - lzo: 693 ms @inode_evictions: 251 .. 250 files - uncompressed: 661 ms @inode_evictions: 252 opensuse leap 15.4 beta - kernel 5.14.21-150400.11 ----- (kvm) .. 250 files - zstd: 811 ms @inode_evictions: 251 .. 250 files - lzo: 912 ms @inode_evictions: 251 .. 250 files - uncompressed: 993 ms @inode_evictions: 251 opensuse tumbleweed ----- kernel 5.14.14 --------------- (kvm) .. 250 files - zstd: 888 ms @inode_evictions: 251 .. 250 files - lzo: 1063 ms @inode_evictions: 251 .. 250 files - uncompressed: 778 ms @inode_evictions: 251 opensuse tumbleweed ----- kernel 5.16.14---------------------- .. 250 files - zstd: 1398 ms @inode_evictions: 250 .. 250 files - lzo: 1323 ms @inode_evictions: 250 .. 250 files - uncompressed: 1365 ms @inode_evictions: 250 Load test results (x86_64): opensuse leap 15.4 beta - kernel 5.14.21-150400.11 ----- (kvm) .. 50 files - zstd: 261 ms @inode_evictions: 51 .. 50 files - lzo: 256 ms @inode_evictions: 51 .. 50 files - uncompressed: 317 ms @inode_evictions: 51 .. 100 files - zstd: 450 ms @inode_evictions: 101 .. 100 files - lzo: 461 ms @inode_evictions: 101 .. 100 files - uncompressed: 471 ms @inode_evictions: 101 .. 150 files - zstd: 618 ms @inode_evictions: 151 .. 150 files - lzo: 624 ms @inode_evictions: 151 .. 150 files - uncompressed: 612 ms @inode_evictions: 151 .. 200 files - zstd: 822 ms @inode_evictions: 201 .. 200 files - lzo: 933 ms @inode_evictions: 201 .. 200 files - uncompressed: 747 ms @inode_evictions: 201 .. 250 files - zstd: 1128 ms @inode_evictions: 251 .. 250 files - lzo: 974 ms @inode_evictions: 251 .. 250 files - uncompressed: 936 ms @inode_evictions: 251 .. 1000 files - zstd: 3517 ms @inode_evictions: 1001 .. 1000 files - lzo: 4373 ms @inode_evictions: 1001 .. 1000 files - uncompressed: 3797 ms @inode_evictions: 1001 ubuntu jammy jellyfish -- kernel 5.15.0.23 - (5.15.27) - (kvm) .. 50 files - zstd: 424 ms @inode_evictions: 1275 .. 50 files - lzo: 423 ms @inode_evictions: 1275 .. 50 files - uncompressed: 207 ms @inode_evictions: 99 .. 100 files - zstd: 1744 ms @inode_evictions: 5050 .. 100 files - lzo: 1838 ms @inode_evictions: 5050 .. 100 files - uncompressed: 373 ms @inode_evictions: 199 .. 150 files - zstd: 4785 ms @inode_evictions: 11325 .. 150 files - lzo: 4660 ms @inode_evictions: 11325 .. 150 files - uncompressed: 689 ms @inode_evictions: 299 .. 200 files - zstd: 9763 ms @inode_evictions: 20100 .. 200 files - lzo: 10106 ms @inode_evictions: 20100 .. 200 files - uncompressed: 938 ms @inode_evictions: 399 .. 250 files - zstd: 17550 ms @inode_evictions: 31375 .. 250 files - lzo: 17337 ms @inode_evictions: 31375 .. 250 files - uncompressed: 1373 ms @inode_evictions: 499 .. 1000 files - zstd: 143614 ms @inode_evictions: 101132 .. 1000 files - lzo: 146724 ms @inode_evictions: 100314 .. 1000 files - uncompressed: 7735 ms @inode_evictions: 1999 Load test results comparisson for compressed files (x86_64): ubuntu jammy jellyfish - compared to - opensuse leap 15.4 beta 50 files gives aprox. 1.6 x more time and aprox. 25 x more inode evictions 100 files gives aprox. 3.8 x more time and aprox. 50 x more inode evictions 150 files gives aprox. 7.4 x more time and aprox. 75 x more inode evictions 200 files gives aprox. 10.8 x more time and aprox. 100 x more inode evictions 250 files gives aprox. 15.5 x more time and aprox. 125 x more inode evictions 1000 files gives aprox. 33.5 x more time and aprox. 100 x more inode evictions Load test results comparisson for uncompressed files (x86_64): ubuntu jammy jellyfish - compared to - opensuse leap 15.4 beta 50 files gives aprox. 0.6 x more time and aprox. 2 x more inode evictions 100 files gives aprox. 0.8 x more time and aprox. 2 x more inode evictions 150 files gives aprox. 1.1 x more time and aprox. 2 x more inode evictions 200 files gives aprox. 1.2 x more time and aprox. 2 x more inode evictions 250 files gives aprox. 1.4 x more time and aprox. 2 x more inode evictions 1000 files gives aprox. 2.0 x more time and aprox. 2 x more inode evictions CPU usage results: The regression causes significant CPU usage by the kernel. ubuntu jammy jellyfish -- kernel 5.15.0.23 (5.15.27) - (kvm) .. 1000 files - zstd: 137841 ms real 2m17,881s user 0m 1,704s sys 2m11,937s .. 1000 files - lzo: 135456 ms real 2m15,478s user 0m 1,805s sys 2m 9,758s .. 1000 files - uncompressed: 7496 ms real 0m 7,517s user 0m 1,386s sys 0m 4,899s Test system specification: host: AMD FX-8370E 8 cores / 8GB RAM / ssd guests (kvm): 2 cores / 2G RAM / ssd test storage medium: RAM disk block device (host and guest) TIA, Bruno
