Hi Veronika, On Fri, Sep 04, 2020 at 07:06:25AM -0400, Veronika Kabatova wrote: > > > ----- Original Message ----- > > From: "Ming Lei" <ming.lei@xxxxxxxxxx> > > To: "CKI Project" <cki-project@xxxxxxxxxx> > > Cc: linux-block@xxxxxxxxxxxxxxx, axboe@xxxxxxxxx, "Changhui Zhong" <czhong@xxxxxxxxxx> > > Sent: Friday, September 4, 2020 3:02:33 AM > > Subject: Re: 💥 PANICKED: Test report for kernel 5.9.0-rc3-020ad03.cki (block) > > > > On Thu, Sep 03, 2020 at 05:07:57PM -0000, CKI Project wrote: > > > > > > Hello, > > > > > > We ran automated tests on a recent commit from this kernel tree: > > > > > > Kernel repo: > > > https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git > > > Commit: 020ad0333b03 - Merge branch 'for-5.10/block' into > > > for-next > > > > > > The results of these automated tests are provided below. > > > > > > Overall result: FAILED (see details below) > > > Merge: OK > > > Compile: OK > > > Tests: PANICKED > > > > > > All kernel binaries, config files, and logs are available for download > > > here: > > > > > > https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawarehouse/2020/09/02/613166 > > > > > > One or more kernel tests failed: > > > > > > ppc64le: > > > 💥 storage: software RAID testing > > > > > > aarch64: > > > 💥 storage: software RAID testing > > > > > > x86_64: > > > 💥 storage: software RAID testing > > > > > > We hope that these logs can help you find the problem quickly. For the full > > > detail on our testing procedures, please scroll to the bottom of this > > > message. > > > > > > Please reply to this email if you have any questions about the tests that > > > we > > > ran or if you have any suggestions on how to make future tests more > > > effective. > > > > > > ,-. ,-. > > > ( C ) ( K ) Continuous > > > `-',-.`-' Kernel > > > ( I ) Integration > > > `-' > > > ______________________________________________________________________________ > > > > > > Compile testing > > > --------------- > > > > > > We compiled the kernel for 4 architectures: > > > > > > aarch64: > > > make options: make -j30 INSTALL_MOD_STRIP=1 targz-pkg > > > > > > ppc64le: > > > make options: make -j30 INSTALL_MOD_STRIP=1 targz-pkg > > > > > > s390x: > > > make options: make -j30 INSTALL_MOD_STRIP=1 targz-pkg > > > > > > x86_64: > > > make options: make -j30 INSTALL_MOD_STRIP=1 targz-pkg > > > > > > > > > > > > Hardware testing > > > ---------------- > > > We booted each kernel and ran the following tests: > > > > > > aarch64: > > > Host 1: > > > ✅ Boot test > > > ✅ ACPI table test > > > ✅ LTP > > > ✅ Loopdev Sanity > > > ✅ Memory function: memfd_create > > > ✅ AMTU (Abstract Machine Test Utility) > > > ✅ Ethernet drivers sanity > > > ✅ storage: SCSI VPD > > > 🚧 ✅ CIFS Connectathon > > > 🚧 ✅ POSIX pjd-fstest suites > > > > > > Host 2: > > > > > > ⚡ Internal infrastructure issues prevented one or more tests (marked > > > with ⚡⚡⚡) from running on this architecture. > > > This is not the fault of the kernel that was tested. > > > > > > ⚡⚡⚡ Boot test > > > ⚡⚡⚡ xfstests - ext4 > > > ⚡⚡⚡ xfstests - xfs > > > ⚡⚡⚡ storage: software RAID testing > > > ⚡⚡⚡ stress: stress-ng > > > 🚧 ⚡⚡⚡ xfstests - btrfs > > > 🚧 ⚡⚡⚡ Storage blktests > > > > > > Host 3: > > > ✅ Boot test > > > ✅ xfstests - ext4 > > > ✅ xfstests - xfs > > > 💥 storage: software RAID testing > > > ⚡⚡⚡ stress: stress-ng > > > 🚧 ⚡⚡⚡ xfstests - btrfs > > > 🚧 ⚡⚡⚡ Storage blktests > > > > > > ppc64le: > > > Host 1: > > > ✅ Boot test > > > 🚧 ✅ kdump - sysrq-c > > > > > > Host 2: > > > ✅ Boot test > > > ✅ xfstests - ext4 > > > ✅ xfstests - xfs > > > 💥 storage: software RAID testing > > > 🚧 ⚡⚡⚡ xfstests - btrfs > > > 🚧 ⚡⚡⚡ Storage blktests > > > > > > Host 3: > > > > > > ⚡ Internal infrastructure issues prevented one or more tests (marked > > > with ⚡⚡⚡) from running on this architecture. > > > This is not the fault of the kernel that was tested. > > > > > > ✅ Boot test > > > ⚡⚡⚡ LTP > > > ⚡⚡⚡ Loopdev Sanity > > > ⚡⚡⚡ Memory function: memfd_create > > > ⚡⚡⚡ AMTU (Abstract Machine Test Utility) > > > ⚡⚡⚡ Ethernet drivers sanity > > > 🚧 ⚡⚡⚡ CIFS Connectathon > > > 🚧 ⚡⚡⚡ POSIX pjd-fstest suites > > > > > > s390x: > > > Host 1: > > > ✅ Boot test > > > ✅ stress: stress-ng > > > 🚧 ✅ Storage blktests > > > > > > Host 2: > > > ✅ Boot test > > > ✅ LTP > > > ✅ Loopdev Sanity > > > ✅ Memory function: memfd_create > > > ✅ AMTU (Abstract Machine Test Utility) > > > ✅ Ethernet drivers sanity > > > 🚧 ✅ CIFS Connectathon > > > 🚧 ✅ POSIX pjd-fstest suites > > > > > > x86_64: > > > Host 1: > > > ✅ Boot test > > > ✅ Storage SAN device stress - qedf driver > > > > > > Host 2: > > > ⏱ Boot test > > > ⏱ Storage SAN device stress - mpt3sas_gen1 > > > > > > Host 3: > > > ✅ Boot test > > > ✅ xfstests - ext4 > > > ✅ xfstests - xfs > > > 💥 storage: software RAID testing > > > ⚡⚡⚡ stress: stress-ng > > > 🚧 ⚡⚡⚡ xfstests - btrfs > > > 🚧 ⚡⚡⚡ Storage blktests > > > > > > Host 4: > > > ✅ Boot test > > > ✅ Storage SAN device stress - lpfc driver > > > > > > Host 5: > > > ✅ Boot test > > > 🚧 ✅ kdump - sysrq-c > > > > > > Host 6: > > > ✅ Boot test > > > ✅ ACPI table test > > > ✅ LTP > > > ✅ Loopdev Sanity > > > ✅ Memory function: memfd_create > > > ✅ AMTU (Abstract Machine Test Utility) > > > ✅ Ethernet drivers sanity > > > ✅ kernel-rt: rt_migrate_test > > > ✅ kernel-rt: rteval > > > ✅ kernel-rt: sched_deadline > > > ✅ kernel-rt: smidetect > > > ✅ storage: SCSI VPD > > > 🚧 ✅ CIFS Connectathon > > > 🚧 ✅ POSIX pjd-fstest suites > > > > > > Host 7: > > > ✅ Boot test > > > ✅ kdump - sysrq-c - megaraid_sas > > > > > > Host 8: > > > ✅ Boot test > > > ✅ Storage SAN device stress - qla2xxx driver > > > > > > Host 9: > > > ⏱ Boot test > > > ⏱ kdump - sysrq-c - mpt3sas_gen1 > > > > > > Test sources: https://gitlab.com/cki-project/kernel-tests > > > > Hello, > > > > Hi Ming, > > first the good news: Both issues detected by LTP and RAID test are > officially gone after the revert. There's some x86_64 testing still > running but the results look good so far! > > > Can you share us the exact commands for setting up xfstests over > > 'software RAID testing' from the above tree? > > > > It's this test (which seeing your @redhat email, you can also trigger > via internal Brew testing if you use the "stor" test set): > > https://gitlab.com/cki-project/kernel-tests/-/tree/master/storage/swraid/trim > > The important part of the test is: > > https://gitlab.com/cki-project/kernel-tests/-/blob/master/storage/swraid/trim/main.sh#L27 > > The test maintainer (Changhui) is cced on this thread in case you need > any help or have questions about the test. > > > > I'll just quickly mention, please be careful if you're planning on > testing LTP/msgstress04 on ppc64le in Beaker, as the conserver overload > is causing issues to lab owners. > > > Let us know if we can help you with something else, I have verified the revised patches does fix kernel oops in 'software RAID storage test'. However, I can't reproduce the OOM in LTP/msgstress04. Could you help to check if LTP/msgstress04 can pass with the following tree(top three patches) which is against the latest for-5.10/block: https://github.com/ming1/linux/commits/v5.9-rc-block-test Thanks, Ming
