On 3/6/20 4:43 PM, CKI Project wrote:
Hello,
We ran automated tests on a recent commit from this kernel tree:
Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
Commit: c302816fcfff - block, bfq: do not insert oom queue into position tree
The results of these automated tests are provided below.
Overall result: FAILED (see details below)
Merge: OK
Compile: OK
Tests: FAILED
All kernel binaries, config files, and logs are available for download here:
https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawarehouse/2020/03/06/474961
One or more kernel tests failed:
s390x:
❌ LTP
❌ stress: stress-ng
Hello, both stress-ng and LTP triggered a panic on s390x, the LTP panic has already been mentioned in
a previous report:
https://lists.linaro.org/pipermail/linux-stable-mirror/2020-March/174549.html
stress-ng: https://cki-artifacts.s3.us-east-2.amazonaws.com/datawarehouse/2020/03/06/474961/s390x_2_console.log
[ 960.031326] mmap: stress-ng-remap (204507) uses deprecated remap_file_pages()
syscall. See Documentation/vm/remap_file_pages.rst.
[ 1100.699451] NET: Registered protocol family 5
[ 1100.899587] NET: Unregistered protocol family 5
[ 1101.021223] NET: Registered protocol family 5
[ 1101.239562] NET: Unregistered protocol family 5
[ 1101.320743] NET: Registered protocol family 5
[ 1101.519533] NET: Unregistered protocol family 5
[ 1101.610575] NET: Registered protocol family 5
[ 1101.849514] NET: Unregistered protocol family 5
[ 1112.850668] sched: DL replenish lagged too much
[-- MARK -- Fri Mar 6 15:25:00 2020]
[ 1314.900059] stress-ng-iomix (353739): drop_caches: 1
[ 1314.900613] stress-ng-iomix (353733): drop_caches: 1
[ 1314.906668] stress-ng-iomix (353709): drop_caches: 1
[ 1315.156617] stress-ng-iomix (353694): drop_caches: 1
[-- MARK -- Fri Mar 6 15:30:01 2020]
[ 1482.676673] ICMPv6: process `stress-ng-procf' is using deprecated sysctl (sys
call) net.ipv6.neigh.enc600.base_reachable_time - use net.ipv6.neigh.enc600.base
_reachable_time_ms instead
[ 1645.499403] list_add corruption. prev->next should be next (00000000310fa210)
, but was 0000000000000000. (prev=0000000000000000).
[ 1645.499435] ------------[ cut here ]------------
[ 1645.499437] kernel BUG at lib/list_debug.c:26!
[ 1645.499499] monitor event: 0040 ilc:2 [#1] SMP
[ 1645.499503] Modules linked in: unix_diag binfmt_misc psnap llc salsa20_generi
c camellia_generic cast6_generic cast_common serpent_generic twofish_generic two
fish_common ofb lrw tgr192 wp512 rmd320 rmd256 rmd160 rmd128 md4 loop tun af_key
crypto_user nfnetlink scsi_transport_iscsi xt_multiport overlay ip6table_securi
ty ip6_tables xt_CONNSECMARK xt_SECMARK xt_state xt_conntrack nf_conntrack nf_de
frag_ipv6 nf_defrag_ipv4 iptable_security ah6 ah4 sctp ghash_s390 prng aes_s390
sunrpc des_s390 libdes qeth_l2 qeth qdio ccwgroup vfio_ccw vfio_mdev mdev vfio_i
ommu_type1 sha512_s390 vfio sha1_s390 ip_tables xfs libcrc32c crc32_vx_s390 sha2
56_s390 sha_common dasd_eckd_mod dasd_mod pkey zcrypt
[ 1645.499548] CPU: 2 PID: 570251 Comm: stress-ng-sysfs Not tainted 5.5.8-c30281
6.cki #1
[ 1645.499557] Hardware name: IBM 2964 N96 400 (z/VM 6.4.0)
[ 1645.499559] Krnl PSW : 0704e00180000000 000000003097bfda (__list_add_valid+0x
a2/0xa8)
[ 1645.499568] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 RI:
0 EA:3
[ 1645.499570] Krnl GPRS: 0000000000000085 00000000312dbbf0 0000000000000075 000
00001ff042a10
[ 1645.499572] 00000001ff051408 0000000000000000 00000001c206ab28 000
0000000000000
[ 1645.499573] 0000000000000000 00000000310fa200 00000000310fa210 000
003e004fbfd00
[ 1645.499575] 00000001fa680000 00000001c206ab40 000000003097bfd6 000
003e004fbfbe8
[ 1645.499581] Krnl Code: 000000003097bfca: c020002f7c46 larl %r2,0000
000030f6b856
[ 1645.499581] 000000003097bfd0: c0e5ffde72e8 brasl %r14,000
000003054a5a0
[ 1645.499581] #000000003097bfd6: af000000 mc 0,0
[ 1645.499581] >000000003097bfda: 0707 bcr 0,%r7
[ 1645.499581] 000000003097bfdc: 0707 bcr 0,%r7
[ 1645.499581] 000000003097bfde: 0707 bcr 0,%r7
[ 1645.499581] 000000003097bfe0: ebeff0880024 stmg %r14,%r1
5,136(%r15)
[ 1645.499581] 000000003097bfe6: b90400ef lgr %r14,%r1
5
[ 1645.499596] Call Trace:
[ 1645.499599] [<000000003097bfda>] __list_add_valid+0xa2/0xa8
[ 1645.499601] ([<000000003097bfd6>] __list_add_valid+0x9e/0xa8)
[ 1645.499606] [<0000000030d41a84>] __mutex_add_waiter+0x3c/0x88
[ 1645.499608] [<0000000030d41e60>] __mutex_lock.isra.0+0xd8/0x508
[ 1645.499611] [<0000000030537364>] psi_show+0x4c/0x1c0
[ 1645.499615] [<000000003075c94c>] seq_read+0xe4/0x4d8
[ 1645.499619] [<000000003072adc4>] vfs_read+0x94/0x160
[ 1645.499620] [<000000003072b1a0>] ksys_read+0x68/0xf8
[ 1645.499623] [<0000000030d45698>] system_call+0xdc/0x2c8
[ 1645.499624] Last Breaking-Event-Address:
[ 1645.499626] [<0000000030d46bd8>] __s390_indirect_jump_r14+0x0/0xc
[ 1645.499628] ---[ end trace 0628f5a603be8e67 ]---
[ 1705.505787] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[ 1705.505800] rcu: 3-....: (4277 ticks this GP) idle=49a/1/0x40000000000000
00 softirq=173963/173963 fqs=2258
[ 1705.505829] (detected by 0, t=6002 jiffies, g=104505, q=612741)
[ 1705.505832] Task dump for CPU 3:
[ 1705.505835] stress-ng-sysfs R running task 0 570254 570245 0x00000006
ltp: https://cki-artifacts.s3.us-east-2.amazonaws.com/datawarehouse/2020/03/06/474961/s390x_1_console.log
[ 6428.638258] LTP: starting proc01 (proc01 -m 128)
[ 6429.364183] ICMPv6: process `proc01' is using deprecated sysctl (syscall) net
.ipv6.neigh.cni-podman0.base_reachable_time - use net.ipv6.neigh.cni-podman0.bas
e_reachable_time_ms instead
[ 6430.215989] LTP: starting read_all_dev (read_all -d /dev -p -q -r 10)
[ 6431.504709] LTP: starting read_all_proc (read_all -d /proc -q -r 10)
[ 6431.547885] ICMPv6: process `read_all' is using deprecated sysctl (syscall) n
et.ipv6.neigh.cni-podman0.base_reachable_time - use net.ipv6.neigh.cni-podman0.b
ase_reachable_time_ms instead
[ 6435.327086] LTP: starting read_all_sys (read_all -d /sys -q -r 10)
[ 6436.214256] list_add corruption. prev->next should be next (00000000d47c6210)
, but was 0000000000000000. (prev=0000000000000000).
[ 6436.214292] ------------[ cut here ]------------
[ 6436.214295] kernel BUG at lib/list_debug.c:26!
[ 6436.214392] monitor event: 0040 ilc:2 [#1] SMP
[ 6436.214397] Modules linked in: kvm dummy minix binfmt_misc nfsv3 nfs_acl nfs
lockd grace fscache sctp rds btrfs blake2b_generic xor zstd_compress raid6_pq zs
td_decompress brd vfat fat loop tun ip6table_nat ip6_tables xt_conntrack iptable
_filter xt_MASQUERADE xt_comment iptable_nat nf_nat nf_conntrack nf_defrag_ipv6
nf_defrag_ipv4 veth bridge stp llc overlay fuse ghash_s390 prng sunrpc aes_s390
des_s390 libdes qeth_l2 qeth qdio ccwgroup vfio_ccw vfio_mdev mdev sha512_s390 v
fio_iommu_type1 sha1_s390 vfio ip_tables xfs libcrc32c crc32_vx_s390 sha256_s390
sha_common dasd_eckd_mod dasd_mod pkey zcrypt
[ 6436.214456] CPU: 0 PID: 729531 Comm: read_all Not tainted 5.5.8-c302816.cki #
1
[ 6436.214460] Hardware name: IBM 2964 N96 400 (z/VM 6.4.0)
[ 6436.217487] Krnl PSW : 0704e00180000000 00000000d4047fda (__list_add_valid+0x
a2/0xa8)
[ 6436.217497] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 RI:
0 EA:3
[ 6436.217500] Krnl GPRS: 0000000000000085 00000000d49a7bf0 0000000000000075 000
00001ff002a10
[ 6436.217501] 00000001ff011408 0000000000000000 00000001b7992a28 000
0000000000000
[ 6436.217503] 0000000000000000 00000000d47c6200 00000000d47c6210 000
003e001bebd00
[ 6436.217504] 00000000fa7e2000 00000001b7992a40 00000000d4047fd6 000
003e001bebbe8
[ 6436.217512] Krnl Code: 00000000d4047fca: c020002f7c46 larl %r2,0000
0000d4637856
[ 6436.217512] 00000000d4047fd0: c0e5ffde72e8 brasl %r14,000
00000d3c165a0
[ 6436.217512] #00000000d4047fd6: af000000 mc 0,0
[ 6436.217512] >00000000d4047fda: 0707 bcr 0,%r7
[ 6436.217512] 00000000d4047fdc: 0707 bcr 0,%r7
[ 6436.217512] 00000000d4047fde: 0707 bcr 0,%r7
[ 6436.217512] 00000000d4047fe0: ebeff0880024 stmg %r14,%r
5,136(%r15)
[ 6436.217512] 00000000d4047fe6: b90400ef lgr %r14,%r1
5
[ 6436.217527] Call Trace:
[ 6436.217530] [<00000000d4047fda>] __list_add_valid+0xa2/0xa8
[ 6436.217532] ([<00000000d4047fd6>] __list_add_valid+0x9e/0xa8)
[ 6436.217538] [<00000000d440da84>] __mutex_add_waiter+0x3c/0x88
[ 6436.217540] [<00000000d440de60>] __mutex_lock.isra.0+0xd8/0x508
[ 6436.217544] [<00000000d3c03364>] psi_show+0x4c/0x1c0
[ 6436.217547] [<00000000d3e2894c>] seq_read+0xe4/0x4d8
[ 6436.217551] [<00000000d3df6dc4>] vfs_read+0x94/0x160
[ 6436.217553] [<00000000d3df71a0>] ksys_read+0x68/0xf8
[ 6436.217556] [<00000000d4411698>] system_call+0xdc/0x2c8
[ 6436.217556] Last Breaking-Event-Address:
[ 6436.217559] [<00000000d4412bd8>] __s390_indirect_jump_r14+0x0/0xc
[ 6436.217561] ---[ end trace 59c684509f54b20b ]---
[ 6496.161797] rcu: INFO: rcu_sched self-detected stall on CPU
[ 6496.161810] rcu: 3-....: (5247 ticks this GP) idle=4aa/1/0x40000000000000
02 softirq=693782/693782 fqs=2851
[ 6496.161838] (t=6001 jiffies g=640657 q=782)
[ 6496.161841] Task dump for CPU 3:
[ 6496.161844] read_all R running task 0 729533 729530 0x00000004
We hope that these logs can help you find the problem quickly. For the full
detail on our testing procedures, please scroll to the bottom of this message.
Please reply to this email if you have any questions about the tests that we
ran or if you have any suggestions on how to make future tests more effective.
,-. ,-.
( C ) ( K ) Continuous
`-',-.`-' Kernel
( I ) Integration
`-'
______________________________________________________________________________
Compile testing
---------------
We compiled the kernel for 4 architectures:
aarch64:
make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg
ppc64le:
make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg
s390x:
make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg
x86_64:
make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg
Hardware testing
----------------
We booted each kernel and ran the following tests:
aarch64:
Host 1:
✅ Boot test
✅ xfstests - ext4
✅ xfstests - xfs
✅ selinux-policy: serge-testsuite
✅ lvm thinp sanity
✅ storage: software RAID testing
🚧 ✅ Storage blktests
Host 2:
✅ Boot test
✅ Podman system integration test - as root
✅ Podman system integration test - as user
✅ LTP
✅ Loopdev Sanity
✅ Memory function: memfd_create
✅ AMTU (Abstract Machine Test Utility)
✅ Networking bridge: sanity
✅ Ethernet drivers sanity
✅ Networking MACsec: sanity
✅ Networking socket: fuzz
✅ Networking sctp-auth: sockopts test
✅ Networking: igmp conformance test
✅ Networking route: pmtu
✅ Networking route_func - local
✅ Networking route_func - forward
✅ Networking TCP: keepalive test
✅ Networking UDP: socket
✅ Networking tunnel: geneve basic test
✅ Networking tunnel: gre basic
✅ L2TP basic test
✅ Networking tunnel: vxlan basic
✅ Networking ipsec: basic netns - transport
✅ Networking ipsec: basic netns - tunnel
✅ audit: audit testsuite test
✅ httpd: mod_ssl smoke sanity
✅ tuned: tune-processes-through-perf
✅ ALSA PCM loopback test
✅ ALSA Control (mixer) Userspace Element test
✅ storage: SCSI VPD
✅ trace: ftrace/tracer
🚧 ✅ CIFS Connectathon
🚧 ✅ POSIX pjd-fstest suites
🚧 ✅ jvm - DaCapo Benchmark Suite
🚧 ✅ jvm - jcstress tests
🚧 ✅ Memory function: kaslr
🚧 ✅ LTP: openposix test suite
🚧 ✅ Networking vnic: ipvlan/basic
🚧 ✅ iotop: sanity
🚧 ✅ Usex - version 1.9-29
🚧 ✅ storage: dm/common
ppc64le:
Host 1:
✅ Boot test
✅ Podman system integration test - as root
✅ Podman system integration test - as user
✅ LTP
✅ Loopdev Sanity
✅ Memory function: memfd_create
✅ AMTU (Abstract Machine Test Utility)
✅ Networking bridge: sanity
✅ Ethernet drivers sanity
✅ Networking MACsec: sanity
✅ Networking socket: fuzz
✅ Networking sctp-auth: sockopts test
✅ Networking route: pmtu
✅ Networking route_func - local
✅ Networking route_func - forward
✅ Networking TCP: keepalive test
✅ Networking UDP: socket
✅ Networking tunnel: geneve basic test
✅ Networking tunnel: gre basic
✅ L2TP basic test
✅ Networking tunnel: vxlan basic
✅ Networking ipsec: basic netns - tunnel
✅ audit: audit testsuite test
✅ httpd: mod_ssl smoke sanity
✅ tuned: tune-processes-through-perf
✅ ALSA PCM loopback test
✅ ALSA Control (mixer) Userspace Element test
✅ trace: ftrace/tracer
🚧 ✅ CIFS Connectathon
🚧 ✅ POSIX pjd-fstest suites
🚧 ✅ jvm - DaCapo Benchmark Suite
🚧 ✅ jvm - jcstress tests
🚧 ✅ Memory function: kaslr
🚧 ✅ LTP: openposix test suite
🚧 ✅ Networking vnic: ipvlan/basic
🚧 ✅ iotop: sanity
🚧 ✅ Usex - version 1.9-29
🚧 ✅ storage: dm/common
Host 2:
✅ Boot test
✅ xfstests - ext4
✅ xfstests - xfs
✅ selinux-policy: serge-testsuite
✅ lvm thinp sanity
✅ storage: software RAID testing
🚧 ✅ IPMI driver test
🚧 ✅ IPMItool loop stress test
🚧 ✅ Storage blktests
s390x:
Host 1:
✅ Boot test
✅ Podman system integration test - as root
✅ Podman system integration test - as user
❌ LTP
⚡⚡⚡ Loopdev Sanity
⚡⚡⚡ Memory function: memfd_create
⚡⚡⚡ Networking bridge: sanity
⚡⚡⚡ Ethernet drivers sanity
⚡⚡⚡ Networking MACsec: sanity
⚡⚡⚡ Networking sctp-auth: sockopts test
⚡⚡⚡ Networking route: pmtu
⚡⚡⚡ Networking route_func - local
⚡⚡⚡ Networking route_func - forward
⚡⚡⚡ Networking TCP: keepalive test
⚡⚡⚡ Networking UDP: socket
⚡⚡⚡ Networking tunnel: geneve basic test
⚡⚡⚡ Networking tunnel: gre basic
⚡⚡⚡ L2TP basic test
⚡⚡⚡ Networking tunnel: vxlan basic
⚡⚡⚡ Networking ipsec: basic netns - transport
⚡⚡⚡ Networking ipsec: basic netns - tunnel
⚡⚡⚡ audit: audit testsuite test
⚡⚡⚡ httpd: mod_ssl smoke sanity
⚡⚡⚡ tuned: tune-processes-through-perf
⚡⚡⚡ trace: ftrace/tracer
🚧 ⚡⚡⚡ CIFS Connectathon
🚧 ⚡⚡⚡ POSIX pjd-fstest suites
🚧 ⚡⚡⚡ jvm - DaCapo Benchmark Suite
🚧 ⚡⚡⚡ jvm - jcstress tests
🚧 ⚡⚡⚡ Memory function: kaslr
🚧 ⚡⚡⚡ LTP: openposix test suite
🚧 ⚡⚡⚡ Networking vnic: ipvlan/basic
🚧 ⚡⚡⚡ iotop: sanity
🚧 ⚡⚡⚡ Usex - version 1.9-29
🚧 ⚡⚡⚡ storage: dm/common
Host 2:
✅ Boot test
✅ selinux-policy: serge-testsuite
❌ stress: stress-ng
🚧 ⚡⚡⚡ Storage blktests
x86_64:
Host 1:
✅ Boot test
✅ Storage SAN device stress - megaraid_sas
Host 2:
✅ Boot test
✅ xfstests - ext4
✅ xfstests - xfs
✅ selinux-policy: serge-testsuite
✅ lvm thinp sanity
✅ storage: software RAID testing
✅ stress: stress-ng
🚧 ✅ IOMMU boot test
🚧 ✅ IPMI driver test
🚧 ✅ IPMItool loop stress test
🚧 ✅ Storage blktests
Host 3:
✅ Boot test
✅ Storage SAN device stress - mpt3sas driver
Host 4:
✅ Boot test
✅ Podman system integration test - as root
✅ Podman system integration test - as user
✅ LTP
✅ Loopdev Sanity
✅ Memory function: memfd_create
✅ AMTU (Abstract Machine Test Utility)
✅ Networking bridge: sanity
✅ Ethernet drivers sanity
✅ Networking MACsec: sanity
✅ Networking socket: fuzz
✅ Networking sctp-auth: sockopts test
✅ Networking: igmp conformance test
✅ Networking route: pmtu
✅ Networking route_func - local
✅ Networking route_func - forward
✅ Networking TCP: keepalive test
✅ Networking UDP: socket
✅ Networking tunnel: geneve basic test
✅ Networking tunnel: gre basic
✅ L2TP basic test
✅ Networking tunnel: vxlan basic
✅ Networking ipsec: basic netns - transport
✅ Networking ipsec: basic netns - tunnel
✅ audit: audit testsuite test
✅ httpd: mod_ssl smoke sanity
✅ tuned: tune-processes-through-perf
✅ pciutils: sanity smoke test
✅ ALSA PCM loopback test
✅ ALSA Control (mixer) Userspace Element test
✅ storage: SCSI VPD
✅ trace: ftrace/tracer
🚧 ✅ CIFS Connectathon
🚧 ✅ POSIX pjd-fstest suites
🚧 ✅ jvm - DaCapo Benchmark Suite
🚧 ✅ jvm - jcstress tests
🚧 ✅ Memory function: kaslr
🚧 ✅ LTP: openposix test suite
🚧 ✅ Networking vnic: ipvlan/basic
🚧 ✅ iotop: sanity
🚧 ✅ Usex - version 1.9-29
🚧 ✅ storage: dm/common
Test sources: https://github.com/CKI-project/tests-beaker
💚 Pull requests are welcome for new tests or improvements to existing tests!
Waived tests
------------
If the test run included waived tests, they are marked with 🚧. Such tests are
executed but their results are not taken into account. Tests are waived when
their results are not reliable enough, e.g. when they're just introduced or are
being fixed.
Testing timeout
---------------
We aim to provide a report within reasonable timeframe. Tests that haven't
finished running yet are marked with ⏱.
