Hello, we hit a kernel panic on ppc64le. We hit this only once, but would like to let you know ASAP. In case we hit this panic again we let you know. The console log is: https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/datawarehouse-public/2021/05/24/308514234/build_ppc64le_redhat%3A1288671967/tests/10038550_ppc64le_1_console.log [10264.732114] restraintd[843]: ** Installing dependencies [-- MARK -- Mon May 24 16:50:00 2021] [10281.612099] BUG: Unable to handle kernel data access on read at 0x302d6d766b2d3870 [10281.612613] Faulting instruction address: 0xc0000000004d39b8 [10281.612809] Oops: Kernel access of bad area, sig: 11 [#1] [10281.612923] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries [10281.613172] Modules linked in: md4 cmac cifs libdes libarc4 dns_resolver nls_utf8 isofs kvm_pr kvm snd_seq_dummy dummy veth minix binfmt_misc can_raw can nfsv3 nfs_acl nfs lockd grace fscache netfs rds tun brd overlay exfat vfat fat loop n_gsm pps_ldisc ppp_synctty mkiss ax25 ppp_async ppp_generic serport slcan slip slhc snd_hrtimer snd_seq snd_seq_device sctp ip6_udp_tunnel udp_tunnel snd_timer snd soundcore authenc pcrypt crypto_user sha3_generic n_hdlc rfkill sunrpc joydev pseries_rng virtio_net net_failover failover virtio_balloon virtio_console crct10dif_vpmsum drm drm_panel_orientation_quirks i2c_core fuse zram ip_tables xfs vmx_crypto crc32c_vpmsum virtio_blk [last unloaded: ltp_insmod01] [10281.615858] CPU: 10 PID: 420920 Comm: dnf Tainted: G OE 5.13.0-rc3 #1 [10281.616024] NIP: c0000000004d39b8 LR: c0000000004d39b8 CTR: c0000000003feda0 [10281.616186] REGS: c000000034f0b4e0 TRAP: 0380 Tainted: G OE (5.13.0-rc3) [10281.616368] MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE> CR: 44224444 XER: 20000000 [10281.616570] CFAR: c0000000004d38a8 IRQMASK: 1 [10281.616570] GPR00: c0000000004d39b8 c000000034f0b780 c000000001e30e00 0000000000000001 [10281.616570] GPR04: 0000000000000000 00000000c0000000 0000000000000001 000000000000000a [10281.616570] GPR08: 00000000bddb0000 0000000000000000 000000000006e12c c008000001a36898 [10281.616570] GPR12: c0000000003feda0 c00000003fff1a80 0000000000000003 c00000000ac0fe40 [10281.616570] GPR16: 0000000000000000 0000000000200000 0000000000008000 0000000000000000 [10281.616570] GPR20: 0000000000000000 ffffffffffffffff c000000097a5e800 c00000000b7e4400 [10281.616570] GPR24: 302d6d766b2d3870 c000000003010c00 c00c00000025e940 0000000000000000 [10281.616570] GPR28: 0000000000000001 0000000000000808 302d6d766b2d3870 c0000000bf58e780 [10281.617956] NIP [c0000000004d39b8] refill_obj_stock+0x68/0x180 [10281.618215] LR [c0000000004d39b8] refill_obj_stock+0x68/0x180 [10281.618354] Call Trace: [10281.618418] [c000000034f0b780] [c0000000004d39b8] refill_obj_stock+0x68/0x180 (unreliable) [10281.618611] [c000000034f0b7d0] [c0000000004a6694] kfree+0x454/0x5c0 [10281.618752] [c000000034f0b840] [c0000000003fee04] kvfree+0x64/0x80 [10281.618901] [c000000034f0b870] [c0080000019f74b8] xfs_log_commit_cil+0x170/0xa70 [xfs] [10281.619316] [c000000034f0b980] [c0080000019ed718] __xfs_trans_commit+0xb0/0x3a0 [xfs] [10281.619595] [c000000034f0b9e0] [c0080000019dd76c] xfs_rename+0x8a4/0xbb0 [xfs] [10281.619876] [c000000034f0bae0] [c0080000019d4534] xfs_vn_rename+0xec/0x1a0 [xfs] [10281.620169] [c000000034f0bb60] [c00000000051cbb0] vfs_rename+0x620/0xa70 [10281.620340] [c000000034f0bc40] [c00000000051ff0c] do_renameat2+0x40c/0x600 [10281.620491] [c000000034f0bd80] [c0000000005202a0] sys_rename+0x60/0x80 [10281.620629] [c000000034f0bdb0] [c00000000002c394] system_call_exception+0x104/0x2c0 [10281.620796] [c000000034f0be10] [c00000000000d45c] system_call_common+0xec/0x278 [10281.620964] --- interrupt: c00 at 0x7fff9dbd4e90 [10281.621077] NIP: 00007fff9dbd4e90 LR: 00007fff9b7100b4 CTR: 0000000000000000 [10281.621239] REGS: c000000034f0be80 TRAP: 0c00 Tainted: G OE (5.13.0-rc3) [10281.621406] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 28284880 XER: 00000000 [10281.621629] IRQMASK: 0 [10281.621629] GPR00: 0000000000000026 00007fffdda0fd10 00007fff9dd87100 000001000261bf70 [10281.621629] GPR04: 000001000597eae0 0000000000000000 0000000000000000 0000000000000000 [10281.621629] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [10281.621629] GPR12: 0000000000000000 00007fff9e22c300 00007fff9b76d1a8 0000000000000000 [10281.621629] GPR16: 00007fff9b76d940 00000100058f9510 00000100033ade20 00007fff9b76da20 [10281.621629] GPR20: 0000010004250c10 00007fff9b76f418 ffffffffffff7fed 000001000427a810 [10281.621629] GPR24: 000001000597eae0 0000000000000000 00007fff9b76a610 ffffffffffffffff [10281.621629] GPR28: 0000000000000001 000001000261bf70 000001000597eae0 00007fffdda0fd90 [10281.622973] NIP [00007fff9dbd4e90] 0x7fff9dbd4e90 [10281.623088] LR [00007fff9b7100b4] 0x7fff9b7100b4 [10281.623200] --- interrupt: c00 [10281.623282] Instruction dump: [10281.623366] 9b8d0938 e92d0030 3fe2ff9b 3bffd980 7fff4a14 e93f0010 7c291800 418200bc [10281.623539] fbc10040 7c7e1b78 7fe3fb78 4bfffe65 <e93e0000> 712a0003 408200f0 886d0938 [10281.623741] ---[ end trace cf4f01c868f7fdda ]--- [10281.626122] [10281.637432] ------------[ cut here ]------------ [10281.637597] WARNING: CPU: 10 PID: 420920 at fs/xfs/xfs_aops.c:491 xfs_vm_writepages+0x108/0x150 [xfs] [10281.638007] Modules linked in: md4 cmac cifs libdes libarc4 dns_resolver nls_utf8 isofs kvm_pr kvm snd_seq_dummy dummy veth minix binfmt_misc can_raw can nfsv3 nfs_acl nfs lockd grace fscache netfs rds tun brd overlay exfat vfat fat loop n_gsm pps_ldisc ppp_synctty mkiss ax25 ppp_async ppp_generic serport slcan slip slhc snd_hrtimer snd_seq snd_seq_device sctp ip6_udp_tunnel udp_tunnel snd_timer snd soundcore authenc pcrypt crypto_user sha3_generic n_hdlc rfkill sunrpc joydev pseries_rng virtio_net net_failover failover virtio_balloon virtio_console crct10dif_vpmsum drm drm_panel_orientation_quirks i2c_core fuse zram ip_tables xfs vmx_crypto crc32c_vpmsum virtio_blk [last unloaded: ltp_insmod01] [10281.639397] CPU: 10 PID: 420920 Comm: dnf Tainted: G D OE 5.13.0-rc3 #1 [10281.639566] NIP: c0080000019b1d30 LR: c0080000019b1c68 CTR: 0000000000000000 [10281.639737] REGS: c000000034f0ad20 TRAP: 0700 Tainted: G D OE (5.13.0-rc3) [10281.639903] MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 84222220 XER: 20000000 [10281.640122] CFAR: c0080000019b1c78 IRQMASK: 0 [10281.640122] GPR00: 0000000000000004 c000000034f0afc0 c008000001a70b00 c000000034f0aff0 [10281.640122] GPR04: 0000000000000000 0000000000000000 c000000034f0b050 0000000000000000 [10281.640122] GPR08: 00000000000000ff c00000005e0ec700 0000000000000000 c008000001a36618 [10281.640122] GPR12: c0000000000b7d48 c00000003fff1a80 0000000000000003 c00000000ac0fe40 [10281.640122] GPR16: 0000000000000000 0000000000200000 0000000000008000 0000000000000000 [10281.640122] GPR20: 0000000000000000 ffffffffffffffff c000000097a5e800 c00000000b7e4400 [10281.640122] GPR24: 302d6d766b2d3870 c000000003010c00 0000000000000000 c00000007b925f40 [10281.640122] GPR28: c000000010e03bd0 c000000034f0b100 c000000053df8298 c000000053df8298 [10281.641614] NIP [c0080000019b1d30] xfs_vm_writepages+0x108/0x150 [xfs] [10281.641876] LR [c0080000019b1c68] xfs_vm_writepages+0x40/0x150 [xfs] [10281.642117] Call Trace: [10281.642177] [c000000034f0afc0] [c000000034f0b070] 0xc000000034f0b070 (unreliable) [10281.642344] [c000000034f0b070] [c0000000003d9564] do_writepages+0x54/0x140 [10281.642495] [c000000034f0b0e0] [c0000000003cad68] __filemap_fdatawrite_range+0x108/0x180 [10281.642669] [c000000034f0b180] [c0080000019daddc] xfs_release+0x254/0x340 [xfs] [10281.643007] [c000000034f0b1d0] [c0080000019c06bc] xfs_file_release+0x24/0x40 [xfs] [10281.643366] [c000000034f0b1f0] [c0000000005049b0] __fput+0xc0/0x340 [10281.643508] [c000000034f0b240] [c000000000180974] task_work_run+0xf4/0x170 [10281.643654] [c000000034f0b290] [c000000000154824] do_exit+0x424/0xca0 [10281.643802] [c000000034f0b350] [c0000000000261d0] oops_end+0x1b0/0x1e0 [10281.643955] [c000000034f0b3d0] [c000000000089ddc] __bad_page_fault+0x174/0x198 [10281.644129] [c000000034f0b440] [c00000000009299c] do_bad_slb_fault+0x9c/0x170 [10281.644299] [c000000034f0b470] [c000000000008c08] data_access_slb_common_virt+0x198/0x1a0 [10281.644469] --- interrupt: 380 at refill_obj_stock+0x68/0x180 [10281.644609] NIP: c0000000004d39b8 LR: c0000000004d39b8 CTR: c0000000003feda0 [10281.644781] REGS: c000000034f0b4e0 TRAP: 0380 Tainted: G D OE (5.13.0-rc3) [10281.644993] MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE> CR: 44224444 XER: 20000000 [10281.645191] CFAR: c0000000004d38a8 IRQMASK: 1 [10281.645191] GPR00: c0000000004d39b8 c000000034f0b780 c000000001e30e00 0000000000000001 [10281.645191] GPR04: 0000000000000000 00000000c0000000 0000000000000001 000000000000000a [10281.645191] GPR08: 00000000bddb0000 0000000000000000 000000000006e12c c008000001a36898 [10281.645191] GPR12: c0000000003feda0 c00000003fff1a80 0000000000000003 c00000000ac0fe40 [10281.645191] GPR16: 0000000000000000 0000000000200000 0000000000008000 0000000000000000 [10281.645191] GPR20: 0000000000000000 ffffffffffffffff c000000097a5e800 c00000000b7e4400 [10281.645191] GPR24: 302d6d766b2d3870 c000000003010c00 c00c00000025e940 0000000000000000 [10281.645191] GPR28: 0000000000000001 0000000000000808 302d6d766b2d3870 c0000000bf58e780 [10281.648552] NIP [c0000000004d39b8] refill_obj_stock+0x68/0x180 [10281.648693] LR [c0000000004d39b8] refill_obj_stock+0x68/0x180 [10281.648835] --- interrupt: 380 [10281.648921] [c000000034f0b7d0] [c0000000004a6694] kfree+0x454/0x5c0 [10281.649064] [c000000034f0b840] [c0000000003fee04] kvfree+0x64/0x80 [10281.649205] [c000000034f0b870] [c0080000019f74b8] xfs_log_commit_cil+0x170/0xa70 [xfs] [10281.649524] [c000000034f0b980] [c0080000019ed718] __xfs_trans_commit+0xb0/0x3a0 [xfs] [10281.649812] [c000000034f0b9e0] [c0080000019dd76c] xfs_rename+0x8a4/0xbb0 [xfs] [10281.650147] [c000000034f0bae0] [c0080000019d4534] xfs_vn_rename+0xec/0x1a0 [xfs] [10281.650496] [c000000034f0bb60] [c00000000051cbb0] vfs_rename+0x620/0xa70 [10281.650640] [c000000034f0bc40] [c00000000051ff0c] do_renameat2+0x40c/0x600 [10281.650784] [c000000034f0bd80] [c0000000005202a0] sys_rename+0x60/0x80 [10281.650933] [c000000034f0bdb0] [c00000000002c394] system_call_exception+0x104/0x2c0 [10281.651101] [c000000034f0be10] [c00000000000d45c] system_call_common+0xec/0x278 [10281.651269] --- interrupt: c00 at 0x7fff9dbd4e90 [10281.651435] NIP: 00007fff9dbd4e90 LR: 00007fff9b7100b4 CTR: 0000000000000000 [10281.651602] REGS: c000000034f0be80 TRAP: 0c00 Tainted: G D OE (5.13.0-rc3) [10281.651773] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 28284880 XER: 00000000 [10281.651979] IRQMASK: 0 [10281.651979] GPR00: 0000000000000026 00007fffdda0fd10 00007fff9dd87100 000001000261bf70 [10281.651979] GPR04: 000001000597eae0 0000000000000000 0000000000000000 0000000000000000 [10281.651979] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [10281.651979] GPR12: 0000000000000000 00007fff9e22c300 00007fff9b76d1a8 0000000000000000 [10281.651979] GPR16: 00007fff9b76d940 00000100058f9510 00000100033ade20 00007fff9b76da20 [10281.651979] GPR20: 0000010004250c10 00007fff9b76f418 ffffffffffff7fed 000001000427a810 [10281.651979] GPR24: 000001000597eae0 0000000000000000 00007fff9b76a610 ffffffffffffffff [10281.651979] GPR28: 0000000000000001 000001000261bf70 000001000597eae0 00007fffdda0fd90 [10281.653338] NIP [00007fff9dbd4e90] 0x7fff9dbd4e90 [10281.653455] LR [00007fff9b7100b4] 0x7fff9b7100b4 [10281.653566] --- interrupt: c00 [10281.653648] Instruction dump: [10281.653741] eb81ffe0 eba1ffe8 ebe1fff8 7c0803a6 4e800020 60000000 60000000 60420000 [10281.653920] 48085119 e8410018 4bffffac 60420000 <0fe00000> 382100b0 38600000 e8010010 [10281.654102] ---[ end trace cf4f01c868f7fddb ]--- On Tue, May 25, 2021 at 6:21 PM CKI Project <cki-project@xxxxxxxxxx> wrote: > > > Hello, > > We ran automated tests on a recent commit from this kernel tree: > > Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git > Commit: cae7e156d77e - Merge branch 'for-5.14/drivers' into for-next > > The results of these automated tests are provided below. > > Overall result: FAILED (see details below) > Merge: OK > Compile: OK > Tests: PANICKED > > All kernel binaries, config files, and logs are available for download here: > > https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/index.html?prefix=datawarehouse-public/2021/05/24/308514234 > > One or more kernel tests failed: > > ppc64le: > Memory function: memfd_create > > aarch64: > ❌ LTP > Storage blktests > > x86_64: > ❌ LTP > > We hope that these logs can help you find the problem quickly. For the full > detail on our testing procedures, please scroll to the bottom of this message. > > Please reply to this email if you have any questions about the tests that we > ran or if you have any suggestions on how to make future tests more effective. > > ,-. ,-. > ( C ) ( K ) Continuous > `-',-.`-' Kernel > ( I ) Integration > `-' > ______________________________________________________________________________ > > Compile testing > --------------- > > We compiled the kernel for 4 architectures: > > aarch64: > make options: make -j24 INSTALL_MOD_STRIP=1 targz-pkg > > ppc64le: > make options: make -j24 INSTALL_MOD_STRIP=1 targz-pkg > > s390x: > make options: make -j24 INSTALL_MOD_STRIP=1 targz-pkg > > x86_64: > make options: make -j24 INSTALL_MOD_STRIP=1 targz-pkg > > > > Hardware testing > ---------------- > We booted each kernel and ran the following tests: > > aarch64: > Host 1: > ✅ Boot test > ✅ ACPI table test > ❌ LTP > ✅ CIFS Connectathon > ✅ POSIX pjd-fstest suites > ✅ Loopdev Sanity > ✅ Memory: fork_mem > ✅ Memory function: memfd_create > ✅ AMTU (Abstract Machine Test Utility) > ✅ Ethernet drivers sanity > ✅ storage: SCSI VPD > ✅ xarray-idr-radixtree-test > > Host 2: > > ⚡ Internal infrastructure issues prevented one or more tests (marked > with ⚡⚡⚡) from running on this architecture. > This is not the fault of the kernel that was tested. > > ⚡⚡⚡ Boot test > ⚡⚡⚡ xfstests - ext4 > ⚡⚡⚡ xfstests - xfs > ⚡⚡⚡ storage: software RAID testing > ⚡⚡⚡ Storage: swraid mdadm raid_module test > ⚡⚡⚡ xfstests - btrfs > ⚡⚡⚡ Storage blktests > ⚡⚡⚡ Storage block - filesystem fio test > ⚡⚡⚡ Storage block - queue scheduler test > ⚡⚡⚡ Storage nvme - tcp > ⚡⚡⚡ Storage: lvm device-mapper test > ⚡⚡⚡ stress: stress-ng > > Host 3: > ✅ Boot test > ✅ xfstests - ext4 > ✅ xfstests - xfs > ✅ storage: software RAID testing > ✅ Storage: swraid mdadm raid_module test > ✅ xfstests - btrfs > Storage blktests > ⚡⚡⚡ Storage block - filesystem fio test > ⚡⚡⚡ Storage block - queue scheduler test > ⚡⚡⚡ Storage nvme - tcp > ⚡⚡⚡ Storage: lvm device-mapper test > ⚡⚡⚡ stress: stress-ng > > ppc64le: > Host 1: > ✅ Boot test > ✅ xfstests - ext4 > ✅ xfstests - xfs > ✅ storage: software RAID testing > ✅ Storage: swraid mdadm raid_module test > ✅ xfstests - btrfs > ✅ Storage blktests > ✅ Storage block - filesystem fio test > ✅ Storage block - queue scheduler test > ✅ Storage nvme - tcp > ✅ Storage: lvm device-mapper test > > Host 2: > ✅ Boot test > ✅ LTP > ✅ CIFS Connectathon > ✅ POSIX pjd-fstest suites > ✅ Loopdev Sanity > ✅ Memory: fork_mem > Memory function: memfd_create > ⚡⚡⚡ AMTU (Abstract Machine Test Utility) > ⚡⚡⚡ Ethernet drivers sanity > ⚡⚡⚡ xarray-idr-radixtree-test > > s390x: > Host 1: > > ⚡ Internal infrastructure issues prevented one or more tests (marked > with ⚡⚡⚡) from running on this architecture. > This is not the fault of the kernel that was tested. > > ✅ Boot test > ✅ Storage: swraid mdadm raid_module test > ✅ Storage blktests > ✅ Storage nvme - tcp > ⚡⚡⚡ stress: stress-ng > > Host 2: > ✅ Boot test > ✅ LTP > ✅ CIFS Connectathon > ✅ POSIX pjd-fstest suites > ✅ Loopdev Sanity > ✅ Memory: fork_mem > ✅ Memory function: memfd_create > ✅ AMTU (Abstract Machine Test Utility) > ✅ Ethernet drivers sanity > ❌ xarray-idr-radixtree-test > > x86_64: > Host 1: > ✅ Boot test > ✅ ACPI table test > ❌ LTP > ✅ CIFS Connectathon > ✅ POSIX pjd-fstest suites > ✅ Loopdev Sanity > ✅ Memory: fork_mem > ✅ Memory function: memfd_create > ✅ AMTU (Abstract Machine Test Utility) > ✅ Ethernet drivers sanity > ✅ storage: SCSI VPD > ✅ xarray-idr-radixtree-test > > Host 2: > ✅ Boot test > ✅ Storage SAN device stress - lpfc driver > > Host 3: > ✅ Boot test > ✅ Storage SAN device stress - mpt3sas_gen1 > > Host 4: > ✅ Boot test > ✅ xfstests - ext4 > ✅ xfstests - xfs > ✅ xfstests - nfsv4.2 > ✅ storage: software RAID testing > ✅ Storage: swraid mdadm raid_module test > ✅ xfstests - btrfs > ✅ xfstests - cifsv3.11 > ✅ Storage blktests > ✅ Storage block - filesystem fio test > ✅ Storage block - queue scheduler test > ❌ Storage nvme - tcp > ✅ Storage: lvm device-mapper test > ❌ stress: stress-ng > > Host 5: > ✅ Boot test > ✅ Storage SAN device stress - qla2xxx driver > > Host 6: > ✅ Boot test > ✅ Storage SAN device stress - qedf driver > > Test sources: https://gitlab.com/cki-project/kernel-tests > Pull requests are welcome for new tests or improvements to existing tests! > > Aborted tests > ------------- > Tests that didn't complete running successfully are marked with ⚡⚡⚡. > If this was caused by an infrastructure issue, we try to mark that > explicitly in the report. > > Waived tests > ------------ > If the test run included waived tests, they are marked with . Such tests are > executed but their results are not taken into account. Tests are waived when > their results are not reliable enough, e.g. when they're just introduced or are > being fixed. > > Testing timeout > --------------- > We aim to provide a report within reasonable timeframe. Tests that haven't > finished running yet are marked with ⏱. > >
